The Reward Is a Service: What RLAIF Taught Me About Grading Reasoning

I started Apertus Nash with a question: could I teach a model to think less when it can, and more when it needs to?

The setup was deliberately constrained. One Apertus-v1.5-8B base model, one NVIDIA DGX Spark (GB10, 128 GB unified memory), a few days, and a target domain of mathematics, statistics and economics. Three stages of supervised fine-tuning, then reinforcement learning from AI feedback with GRPO.

What I actually learned had less to do with the policy than with the thing scoring it.

---

The part nobody warns you about

Here is the uncomfortable property of RL on language models: the policy will find whatever your reward forgets to punish, and it will find it in the middle of a run, long after you committed to a rubric.

In stage 4, around update 78, the model drifted into loops. It had discovered that a long, repetitive chain of thought that never quite terminated was not penalised the way a wrong answer was. Truncated answers were sitting at 44.7% of the batch. The reward function I had written before the run started simply did not have an opinion about this failure mode, because I had not imagined it.

The usual options are both bad. Stop the run, edit the reward, restart — and lose the optimiser state and every rollout you have paid for. Or let it finish and write off the compute.

---

Putting the reward behind HTTP

So the reward does not live in the trainer. It lives in its own container, on the other side of an internal HTTP call:

Trainer container

GPU · DGX Spark

policy

Apertus-8B + LoRA

rank 64, alpha 128

rollout

vLLM

8 prompts × 8 samples

update

Dr. GRPO

advantages within each group

POST /reward — 8 answers8 rewards + version hash

Reward service container

CPU · internal network only

step 1 — measured

Exact facts

SymPy grader in its own process, repetition, reply defects

step 2 — judged

AI judge

must trust the facts; scores all 8 in one call

reward.py · reward_prompt.md · reward_config.json
— all three re-read from disk on every request

The service re-reads its rubric and its code on every single request. A fix lands at the next update: no restart, no lost optimiser state, no cold start. Every reward it returns carries a version hash, so the training log records which rubric produced which number. A broken edit keeps the last good version rather than poisoning the run, and a group the judge cannot score is dropped rather than guessed.

Both of the mid-run changes below happened inside a single trainer process that was never restarted:

4.4%

from 44.7%

Truncated answers, after cut-off responses started scoring −2 and repetition was penalised (stage 4, ~update 78).

$0.022

from $0.074

Cost per update, after the judge stopped computing and the service measured the facts instead (stage 5, update 31).

That second one matters for a different reason. I had been asking the judge to compute things — and a model asked to compute is a model given the opportunity to be wrong about ground truth. Moving the arithmetic into the service made it cheaper and removed a class of grading error.

---

Facts first, judgement second

The reward is not "ask a model if the answer is good". It is a layered thing, and the layers are ordered by how much I trust them.

For each group of eight answers, the service first measures what it can measure exactly: a field-by-field comparison against a verified reference (graded with SymPy, in its own process, with a timeout), repetition inside the chain of thought, and structural defects in the reply. Only then does a DeepSeek judge see the answers — and the prompt instructs it to trust those facts and never substitute its own solution for the reference. It scores all eight in one call, which also makes the relative judgement cheaper and more consistent.

Which collapses into:

Note the last two rows. "Addressing the grader" is in there because the model tried it. So is "looping while correct", because being right is not a licence to ramble.

---

What came out

On stage 5, accuracy on the training problems across four epochs went 0.15 → 0.15 → 0.18 → 0.25. Broken down by difficulty:

epoch 1epoch 4

Intermediate problems0.20 → 0.39

Advanced problems0.05 → 0.16

Challenging problems0.00 → 0.00

None of the challenging problems was ever solved. And those are training-set numbers — there is no proper held-out evaluation yet, and I would rather say that plainly than dress them up.

The behavioural change is the more interesting result. Side by side with the base model on the same settings:

base modelApertus Nash

Learning algebra (English)1,152 → 55

Redistribution & welfare (French)1,748 → 930

Nash is far more concise — twenty times fewer thinking tokens on the easy algebra turn — and it thinks longer than its own average only on the genuinely hard request in the set. That was the original question, and the answer is a qualified yes.

It also answered three of the four French turns in English. That is a real weakness, and it comes straight from the SFT mix.

---

The honest summary

This is an experiment, not a finished model. A few days on one DGX Spark was enough for the supervised stages; the RL stage needs considerably more time to generalise beyond its training problems. The SFT mix also needs to cover more domains and restore basic general instruction-following, which the specialisation eroded.

But the architectural idea travels further than the model does. If your reward is a program, treat it like one you will have to debug in production — version it, let it hot-reload, make it fail closed, and keep the exactly-measurable facts out of the judge's hands.

Everything is public: the code and configs, the reward rules in reward_service/reward_prompt.md, the 1,250-row verified dataset, and the merged model.