Understanding RL as a Data and Reward Problem

The core objective was to learn how Reinforcement Learning (RL) is done. Particularly since I learnt that all the massive progress that was made in LLMs is because of RL. Previously, my attempt was to do SFT and then RL on top of that. But for niche use cases, after SFT on small models, I failed to find any need of RL. The details are mentioned in my previous blog. So, I didn't bother to do any SFT this time. I jumped into RL right away. Obviously, this is just to learn and shouldn't be done when you solve a real use case. If you run out of data for SFT and still find real use cases failing in your production, there is a chance RL might fit. The reason I say 'might' is because that too depends on the variance observed in your model output.

This blog is about how to approach a problem where RL 'might' fit. The problem statement I tried is 'PII de-identification' - find every Aadhaar number, PAN, phone number, address, account number and name in a document. The model chosen is Qwen2.5-0.5B-Instruct.

Getting some data which may or may not be useful !

RL is more of a 'data research' problem than a 'training' problem. Moreover, to do data research at scale is a challange. But how do you do this 'data research'. I really don't know, but I did the best approach I could think of -- trial and error !

Idea is simple - have some cheap ways of generating 'real looking' data and then find the best subset suitable to do RL.

Two ways I tried to generate data:

• A synthetic generator that injects known PII into templates styled as KYC forms, bank statements, chats, emails and medical notes. Because the generator places each entity, the ground truth is exact by construction and the supply of labelled data is unlimited and free. This was my primary training data.
• Microsoft's Presidio sentence faker, for evaluation. Its sentence templates, span conventions and entity taxonomy are kept exactly as they are; only the values are regenerated with Faker's en_IN provider, because the bundled records are Czech and Italian names and useless for an Indian-locale test. Sentences are then concatenated into multi-PII documents — the gold carries no character offsets, only (text, type, cluster), so concatenation composes by simple union with nothing to realign.

The split matters. The training documents are mine end to end, which means the scorer and the generator share an author and therefore share every convention. The evaluation documents are somebody else's structure with my values, which is the only arrangement that can surface a convention I got wrong without me noticing.

One deliberate addition to the evaluation set: distractor sentences whose only entities were dropped by the type mapping — organisations, cities, dates. They read as entity-bearing and contribute no gold at all. That is the "looks like PII, is not PII" pressure that my training generator never applies. These were generated using the Presidio sentence faker.

Data distribution

The training data generator has 53 types — recipes, each generating unlimited examples — keyed like govt_id|kyc_form|d4|c1|x0|p6: government IDs, KYC styling, 4 entities, 1 name form, no distractors, 6 units of filler.

The grid is not a full cross product. That would be 1,260 types, most of them uninformative corners. So, it is a bit custom to ensure data generated is real-looking.

Reward Design

RL needs a reward for every attempt. If a human has to read the output, the loop is too slow; if another model judges it, the reward is itself an unreliable model. So the first requirement is narrow and non-negotiable: correctness has to be decidable by a deterministic program.

I decided on a non-binary reward scoring function. It is a few hundred lines of pure functions — no model, no network call, nothing that can be argued with — and it scores every answer from 0 to 1 as a weighted sum of five checks:

Recall outweighs precision because a missed Aadhaar number is a regulatory incident and an over-redacted street name is merely an annoyance.

Potential prevention of reward hacking Emit nothing and recall collapses; emit everything and precision collapses. RL will find a shortcut if one exists, so both have to be unprofitable before training starts. Grounding ensures that the model doesn't make up stuff.

The score is dense, not binary, and that makes it easier to find a useful gradient.

Finding the advantage - is there scope to learn more from RL ?

Next was to see if the base model actually had a scope to improve or already solves our problem. I used standard metrics to quantify this. The samples where we couldn't find a useful gradient is essentially not helpful data. As I mentioned earlier, this is essentially 'trial and error'.

Before going further, few definitions (aka jargon) used in the context of RL

• Environment — for a language model there is no game world or a setup. It is a prompt plus a scoring function: show a document, let the model answer, score the answer.
• Rollout — one complete attempt. Sampled at temperature 1.0, not greedy, because the whole method depends on the same prompt producing different answers on different attempts.
• Group — the k rollouts generated from one prompt. This is the unit that matters. Why groups of K ? That is because of the way GRPO works.
• Policy — the model being trained, understood as a probability distribution over next tokens. This is the policy that we optimise using the Grouped Relative Policy Optimization (GRPO).
• Effective fraction is the share of groups that are useful, and it is the one that says whether RL can learn here at all.
• pass@k asks whether any of the k attempts cleared the bar. This says whether model can give the right answer.
• Mean reward says how well on average was the task ever done properly.

So, to find data where where a useful learning signal exists, I need to setup an environment where I can generate rollouts, score them with a reward function, and compute relative advantage within each groups of K.

The scale of rollouts follows from the grid. Each of the 53 types gets 10 documents, and each document gets k rollouts:

53 types × 10 documents          =    530 prompts
530 prompts × 32 rollouts        = 16,960 rollouts

Two passes were run this way, at k=8 and k=32.

Running the untrained model across that grid shows where the difficulty actually lives:

Density dominates every other axis. Recall falls threefold from one entity to ten, and the chance of a clean answer in eight attempts goes from 97% to 11%. Precision meanwhile rises, from 0.498 to 0.575 (not shown in the graph above) — so the model is not getting noisier under load, it is getting incomplete. For PII redaction use case, that can be dangerous.

Why any of this is learnable

GRPO generates k answers to one prompt, scores them, and computes each answer's advantage as its distance from its own group's mean:

advantage_i = (reward_i − mean(group)) / std(group)

The group is the baseline here.

If all k answers score the same, the spread is zero, every advantage is zero, and that prompt teaches nothing. This type is essentially a useless sample

This is also where the dense reward earns its place. Take a group of eight failures. Under binary scoring: eight zeros, no spread, no advantage. Under partial credit: eight attempts that fail by different amounts, real spread, a usable advantage.

Whether a task is trainable is not purely a property of the model. It is a joint property of the model and the reward you wrote.

Interpretting the base model performance

The first run of the grid produced a mean reward of 0.303, and roughly a quarter of that shortfall had nothing to do with finding PII.

The model has to return JSON. When it returns something that is not JSON the score is 0.0 regardless of content — a gate failure — and it was failing that gate 23% of the time. A quarter of the apparent incompetence was punctuation.

The cheap fix is guided decoding, and it needs no training at all. Rather than sampling freely and hoping, the sampler is constrained at every step to tokens that keep the output matching a grammar. In vLLM that is a guided_decoding regex or JSON schema passed with the sampling parameters; it compiles to a finite-state machine over the vocabulary and masks every token that would break the pattern. Malformed output becomes structurally impossible, if we use this method.

Re-running the identical grid under the constraint:

+0.134 with zero training. That number is worth isolating, because it separates two problems that usually get argued about rather than measured. This is the cheapest reward lift I got in the entire exercise ! Formatting has exactly one correct target shape, which makes it a supervised fine-tuning job — and here it did not even need that much. The remaining 0.563 to the ceiling is capability, which I hope can come out of RL !

Guided decoding is also the better instrument than fixing format by SFT training, because it changes no weights and so cannot leak task knowledge into the model being measured. It is not free at run time — grammar-constrained sampling in vLLM ran at 5,648 tokens/second against 23,543 free-form, a 4.2× throughput penalty — which is fine for a measurement job.

Increasing K can potentially make more of your data useful

The k=8 pass reported 10 of the 53 types as never solved. Essentially their pass@8 scores were 0. The obvious reading is that these are beyond the model, RL has nothing to reinforce, and they should be dropped from training.

So I repeated the pass at k=32. Hypothesis was that maybe there are right answers lying around in the model output distribution, which we cannot capture at k=8.

Eight of the ten flipped to trainable.

Summary of the metric movement when we changed measurement from k=8 to k=32. The effective fraction increase tells you that we got more data than before which are trainable.

The RL scoping verdict: 51 of 53 types carried a usable advantage, with effective fraction at 98–100% almost everywhere, including types solved about 1% of the time.

Training runs

Read as a loop: at each training step, sample 4 prompts and generate 8 constrained rollouts per prompt, producing 32 trajectories. Score each trajectory with the reward function and use GRPO to calculate relative advantages among the eight responses to each prompt. Use those advantages to estimate the policy gradient, while applying a 0.02 KL penalty to discourage excessive deviation from the reference policy. Update the model starting with a learning rate of 2e-6, gradually decay it to zero, and repeat for 150 optimizer steps — yielding about 4,800 total rollouts.

Reward roughly doubled, and held on 11 document types that were never trained on (the held out set). Every reward component rose together — recall eightfold, alongside precision, grounding and coreference — which is the standard check for reward hacking, since a model gaming one term shows that term running away while the others stall.

Reading an RL training run is a bit different from SFT run

Train reward is not a learning curve. In SFT, training loss falls smoothly because every step sees the same kind of example. Here each step draws 4 prompts from types that differ enormously in difficulty, so most of the step-to-step movement is which types were drawn, not whether the model improved. Read it over ~20-step windows, or not at all.

The loss is not the metric. The SFT habit of watching train-loss against eval-loss maps here onto train reward against sealed-eval reward.

The degenerate-group rate is the stopping signal, and it has no SFT analogue at all. The share of groups where all eight rollouts scored identically and therefore produced no gradient. It is exactly 1 − effective fraction, measured live. It climbed from 0.03 to 0.37.

Final results

These numbers are the end of several iterations rather than one run, and the config moved as issues surfaced. Recall and precision started at 0.40/0.25 and were rebalanced to 0.325 each; weight later shifted to type accuracy (0.10 → 0.25) once the component log showed it was the only term still varying. Training settled at 150 steps, because external scores peak around there and decay afterwards.

Measured on 600 documents with 3,214 gold spans across 12 entity types, written by someone else. Strict token F1 requires the right span and the right label; typeless requires only the right span. A 0.5B model went from 0.105 to 0.406 on data it had never seen, for about a dollar of GPU.

Why external precision falls as training continues is the open question. It drops at every checkpoint — 0.391 at step 50 to 0.253 at step 300 — while the reward's own precision component sits near 0.96 and flat throughout, because a document made only of PII contains nothing that should not be extracted. Something about longer training makes the model reach further, and the training signal cannot see it happening. I do not have the answer yet !