A big general-purpose language model can do almost anything, and you pay for that breadth on every single call. The claim I wanted to test is that for one narrow job you can take a much smaller open model, train it a little on your own examples, and end up with something both cheaper to run and better at that one job than the big model. Cheaper is easy to believe. Better is the surprising half.
Training a model further on your own examples is called fine-tuning, and the claim only means anything if you can measure it. So I picked three jobs where a computer — not a human reading the output and nodding — can decide whether an answer is right. I took these machine verifiable use cases to also try and learn how to do Reinforcement Learning with Verifiable Rewards (RLVR). I also thought that these are narrow enough for a small model to learn. The three use cases I tried are:
- Text to PromQL. A question in English goes in ("how much memory is this service using?"), a query in a monitoring language called PromQL (Prometheus Query Language) comes out. You can just run the query and see whether it returns what it should.
- Address cleanup. A messy Indian postal address goes in, tidy labelled fields come out. Since I generated the messy version from the tidy one, I know the right answer exactly.
- Screenshot to HTML. A picture of a web page goes in, the code to rebuild it comes out. Render that code and compare the result to the original picture.
The common shape: a program can score the answer. That is the constraint the whole project rests on, because it means I can measure a change instead of forming an impression of one.
What follows is a brief description of the broad learnings that I had.
The training data, and the part of it I could not see
Training examples came from two places. Where Hugging Face — a public library of datasets and models — already had something close to my task, I started there. Where it did not, I generated examples myself. I thought generating data would be pretty easy given how good LLMs are now-a-days. For addresses that meant starting from the real India Post directory of pincodes and localities, building a clean address, and then deliberately messing it up. That way the clean version is the right answer by construction, for free, and I never have to label anything by hand.
Then I looked at what I had actually produced, and found three problems. None of them showed up as an error. All of them capped how good the model could get.
I accidentally built a village dataset. I picked pincodes uniformly at random from the 19,238 that exist. India has far more villages than cities, so uniform picking gives you mostly villages: Mumbai got the same chance as any hamlet. Across 2,300 generated addresses the big metros turned up about eleven times, and Kochi and Gurugram never appeared at all. Real address traffic is the other way round — it piles up in cities.
One of my mess-up rules did nothing 95.7% of the time. People write city names badly, so one data generation rule replaced a proper name with a sloppy one — Bengaluru becomes Bangalore, or blr, or Banglore. Teaching the model to undo that was close to the whole point. But the rule worked off a hand-written table of ten cities, and if the address's city was not in the table it quietly left the address alone. My dataset was mostly villages. So the table missed almost every time, and the single behaviour I most wanted to teach was present in about 4% of my examples.
My table disagreed with my own directory. I had typed those city names from memory, and the official directory does not use the names I do. It says Ernakulam, not Kochi. Gurgaon, not Gurugram. There is no district called "Delhi" at all — it is nine separate sub-districts. Every name that did not match was a city that silently never appeared.
None of this is visible in the row count, and the row count is what you instinctively check. Four thousand examples sounds like plenty.
What matters is not how many examples you have. It is how many genuinely different situations they contain.
There is a second trap in generating your own data, and I walked into it. After training, the model scored 0.99 on my held-out test examples — near perfect. Then I typed in a real address I copied off a delivery app:
d1301, sattva sanctity, sarjapura road, bengaluru
It misspelled the building name and filed the city under locality. The model was not broken. My generator always wrote addresses in the same field order, used thirteen invented building names, and always included a locality. Real addresses skip fields, arrive in any order, and are full of proper nouns no generator would invent. My test examples came out of the same generator as my training examples, so all that 0.99 ever proved was that the model had learned my simulation of the problem. Fifty real addresses set aside at the start would have exposed this on day one.
Grading the answers, and then grading the grader
To measure anything I need a scorer: an ordinary program that takes the model's answer and the correct answer and returns a number between 0 and 1. No second model is involved in judging — it is just code. For PromQL it runs both queries and compares results. For addresses it compares the fields one by one. For screenshots it compares the rendered page against the original.
Then there is a step I nearly skipped, and it turned out to matter more than anything else in the project. Hand the scorer the correct answer, pretending the model produced it. It has to come back with exactly 1.0. A perfect answer must score perfectly. If it scores 0.87, the scorer itself is wrong, and every number the project produces afterwards is partly measuring that error rather than the model.
I ran this check — gold against gold — on every task, and it found a bug every time. On screenshot-to-HTML task, it found four. Two are worth explaining, because they are the kind of thing that hides forever:
It was comparing where things appear on screen with where they appear in the code. A block on a web page has two different orders: the order you see it in, and the order it is written in the HTML source. CSS can make these disagree — a sidebar can be written second in the source but displayed on the left. My scorer took the model's on-screen order and compared it against the original's source-code order. So a flawless reproduction of a page with a sidebar was marked down for the original page's own layout, scoring 0.429 out of 1.0.
Ties were broken by luck. To compare two pages I sort their blocks by position — down the page first, then across. But two blocks can sit at the same position, and then the sort has to fall back on something else to break the tie. It fell back on whatever order the blocks happened to be sitting in, which was different for the two lists being compared. The result: 117 documents came out "different" from themselves. The fix was to add a final tiebreaker so that identical inputs always sort identically.
The other two were worse in effect. Comparing two empty pages returned 0.55 instead of 1.0. And position was scored on exact pixels, so a layout shifted 40 pixels — indistinguishable to a human — scored 0.117, and at 120 pixels it scored zero. That last one meant a fifth of the available score was impossible for any model to earn, because no model reproduces pixel positions exactly. I would have paid to train against a target that could not be reached, and then concluded the model was incapable.
The address scorer had a bug with a nastier shape. One of my mess-up rules deleted the pincode from the input, but the expected answer still contained it. The only way to score well was to invent a six-digit number that was not there. I was one run away from explicitly training a model to make up postcodes.
Write the scorer first. Then try to break it.
Ask what the base model can already do
The most useful hour I spent was the one where I trained nothing at all. I ran the untouched model — Qwen3.5-0.8B, a small open model — against my scorer, and looked at the results field by field.
The address task has eight fields. Five are copied out of the input: premise (flat or plot
number), building, street, locality, landmark. Three have to be produced in an official
form: city, state, pincode. Alongside those I score grounded, which asks whether the
model stuck to what the input actually said instead of inventing content.
Qwen3.5-0.8B, before any training (300 held-out addresses)
grounded █████████████████████ 1.00 already solved
pincode █████████████████░░░░ 0.82 already solved
landmark █████████████░░░░░░░░ 0.61
field_f1 █████████░░░░░░░░░░░░ 0.44 how well fields match overall
geo █████████░░░░░░░░░░░░ 0.42 do city, state and pincode agree
city █████░░░░░░░░░░░░░░░░ 0.26 ← the actual problem
overall score 0.51
This changed what I built. The model was already refusing to invent things, and already pulling pincodes out of the text 82% of the time — effort spent there could win almost nothing. Turning a sloppy city name into its official form sat at 0.26, and the failures repeated: shown a city it did not recognise, it guessed a familiar big one.
Had I skipped this hour, I would have built a tidy dataset covering all eight fields evenly and spent most of my budget teaching the model things it could already do.
Training, and knowing when to stop
So the training set is not a fair sample of the task. It is deliberately skewed towards the places the model was failing — heavy on unfamiliar city names, light on the pincodes it already handled.
Then you train, and the question becomes when to stop. Two numbers move while training runs. Training loss measures how well the model fits the examples it is being shown; it will keep improving for as long as you let it run, which sounds good and is not — past a point it just means the model is memorising. Eval loss measures the same thing on examples it is not being trained on, which is what you actually care about. So you save the model periodically — each saved copy is a checkpoint — and keep the one where eval loss was lowest.
The textbook picture is a U: eval loss falls, reaches a bottom, then climbs again as the model starts memorising. My run did not do that. Eval loss dropped hard for the first hundred steps and then went flat — 0.4996, 0.4924, 0.4865, 0.4969, 0.4912, 0.4881, 0.4873 — jittering inside a band about a thousandth of a point wide, while training loss carried on down from 0.48 to 0.38. The memorising is real, but you see it as the gap opening between the two lines, not as the eval line turning upwards.
That makes "keep the best checkpoint" a less precise instruction than it sounds. Step 160 was picked as best at 0.4865; step 240 finished at 0.4873. The difference is 0.0008, which is far too small to mean anything. What the curve honestly told me was when to stop — past about step 120 I was spending money for nothing — not which exact checkpoint to ship.
Did it learn the job, or memorise my examples?
This is the question that makes or breaks the whole exercise, and answering it needs the test examples to be held back in the right way. Splitting rows at random is close to useless here, because a random test row was built from the same ingredients as the training rows. So I held out whole categories instead: six entire states appear nowhere in training, not in one single address, and two of the ways of messing up an address were never shown during training either.
Then I compared the trained and untrained model as two arms of the same run, over the same addresses, switching the trained part on and off — so nothing but the training differs.
score
base model, untrained 0.51
trained, familiar territory 0.63 +0.12
trained, mess-up types never seen before 0.64
trained, states never seen before 0.55
Read the last two lines against the first. The trained model handles kinds of mess it was never shown, and places it was never shown, better than the untrained model handled the easy case. It did not simply memorise the cities in its training data — it picked up a method: find the city in the text and rewrite it properly. Had I split rows at random, every one of these lines would have read the same and I would have learned nothing.
The same split also showed me what did not transfer. Broken down by field:
familiar states unseen states
pincode 0.94 0.92 transfers fine
city 0.46 0.34 degrades
geo consistency 0.69 0.47 falls apart
The pincode is sitting right there in the input, so reading it is a skill that works anywhere. Geographic consistency is different: to check that the city, state and pincode agree, the model has to know which state a pincode belongs to. That is not a skill, it is a fact — and facts about states I deliberately withheld were never learned. The model answers confidently and wrong.
Fine-tuning is good at teaching a method the model can apply to whatever is in front of it. It cannot install knowledge you kept from it.
The fix was not more training. Turning a pincode into a state is a lookup in a table I already had in code. It should never have been the model's job.
You run out of data before you run out of problems
Eventually the data stops coming. You have used the public datasets that were close enough to be useful, your generator is producing variations on things it has already produced, and the honest description of what happens next is that you stop looking rather than that you finish.
That is survivable. What is not survivable is trusting the number you happen to be holding at that moment.
The temptation at that point is to run the old model over one batch of examples, the new model over another, compare the two averages and call the difference an improvement. That does not work, because examples vary enormously among themselves — some are simply harder than others — so two averages can differ for reasons that have nothing to do with the models. On a small sample that difference can easily be larger than the effect you are looking for.
The fix costs nothing: run both models over the same examples and compare them one at a time. My eval script ends up doing something stricter still — it loads the model once and switches the trained part on and off between the two arms, so the two runs share not just the same examples but the same weights, the same batching and the same sampling. Nothing differs except the thing being measured.
Then you deploy, and real traffic starts finding cases your generator never imagined — a field order you did not think of, a building name no generator would invent, a question phrased in a way your training set never covered. This is not a failure of the method, it is the method: production is the only source of examples that is not your own simulation of the problem. The useful posture is to expect breakage, keep the machinery that turns each break into a new test case, and stop treating the held-out set as a finish line.
Would Reinforcement Learning have helped?
There are two broad ways to improve a model. Supervised fine-tuning — everything above — shows it the right answer and says "produce this." Reinforcement learning never shows it an answer: it lets the model try, scores the attempts, and pushes it towards whatever scored better. RL is the glamorous one, and I wanted it to work. I tried it on all three tasks. It never once helped, and eventually I understood why.
Reinforcement learning can only strengthen behaviour the model already produces some of the time. It shifts the odds between things the model already does. It cannot add something that was never there.
The algorithm I used, GRPO (DeepSeek fame), makes this very concrete. For one question it generates several answers, scores them all, and works out how much better or worse each was than the average of that group. That comparison is the entire learning signal — and if all the answers in a group score the same, there is nothing to compare, the signal is zero, and that question teaches the model nothing at all.
So before spending anything, ask the untrained model one question: generate eight answers to each prompt and sort every prompt into three piles.
never solved all 8 attempts fail → nothing to reinforce
always solved all 8 attempts succeed → nothing to improve
mixed some succeed, some fail → THE ONLY USEFUL PILE
That middle pile is the entire fuel supply. It can be empty for two opposite reasons: the task is too easy and the model always succeeds, in which case you need harder examples; or the task is too hard and it always fails, in which case you need supervised fine-tuning first, to get the model to the point where it sometimes succeeds.
There is a companion measurement called pass@k: let the model try k times and count the question as solved if any attempt worked. Comparing one attempt against eight tells you how much ability is in there but coming out unreliably.
Twenty-seven points of gap looks like a green light. It is not, and the difference is subtle but decisive: pass@8 counts a question as solved if any of eight attempts worked anywhere across the whole set. GRPO needs something stricter — a single question where some attempts succeed and others fail. Measured that way, on the address task:
8 attempts each, across 200 prompts
never solved 0.995 every attempt failed
always solved 0.000 every attempt succeeded
mixed 0.005 1 prompt in 200
One prompt in two hundred had anything to learn from. Dead on arrival — and I knew it for the price of one round of generating answers, rather than a training run. Fine-tuning first raised that to one in a hundred. Still nothing.
Text-to-PromQL is where I stopped arguing with the result, because there I ran RL anyway, twice, and kept every version:
valid queries correct results
base model 0.33 0.00
RL on the base model 0.33 0.00 ← nothing moved
after fine-tuning 1.00 0.22
RL after fine-tuning 1.00 0.22 ← nothing moved
Two runs, and both landed exactly where they started. From the base model there was no correct behaviour to strengthen. From the fine-tuned model the behaviour was already consistent, so every group of attempts scored alike and there was nothing to compare. The only thing that ever moved these numbers was showing the model the answer.
RL clearly does work — it is behind a great deal of recent progress. But the conditions it needs are specific: a task the model can genuinely already do and merely does unreliably, plus a scorer you trust completely. On small models and narrow, well-specified jobs, I did not find that situation. Supervised fine-tuning was sitting right there, and it worked.
What this was actually about
The hypothesis half held up. On addresses, a 0.8-billion-parameter model went from 0.51 to 0.63 and kept most of that on states it had never seen. On PromQL, fine-tuning took a model that produced valid queries a third of the time to one that produced them every time, and took correct results from zero to a fifth. Both for an hour on a rented GPU, at a running cost nowhere near a frontier API call. Screenshot-to-HTML I never got past the starting line, probably due to every model being very good at coding tasks these days. So there I have a measurement rather than a result.
But almost nothing I learned was about training. Training is a script and an hour of rented compute — a few hundred lines, and the least interesting part of the project.
Every expensive mistake was in the data or the measurement, and every one was silent. Nothing crashed. The curves looked healthy. The numbers looked plausible. A scorer that could not score a perfect answer as perfect. A dataset where the key lesson appeared in 4% of rows. A test set that could only ever measure my own simulation. Not one of them announced itself; each was found only on debugging a bad model output.
Which is the finding, and it is not a subtle one:
In fine-tuning, the datasets are the work. The training data decides what the model can learn, and the evaluation data decides whether you are entitled to believe it. Everything else is plumbing !