This blog is about how to approach a problem where RL 'might' fit. The problem statement I tried is 'PII de-identification' - find every Aadhaar number, PAN, phone number, address, account number and name in a document. The model chosen is Qwen2.5-0.5B-Instruct.

Getting some data which may or may not be useful !

RL is more of a 'data research' problem than a 'training' problem. Moreover, to do data research at scale is a challange. But how do you do this 'data research'. I really don't know, but I did the best approach I could think of -- trial and error !

Idea is simple - have some cheap ways of generating 'real looking' data and then find the best subset suitable to do RL.

Two ways I tried to generate data:

• A synthetic generator that injects known PII into templates styled as KYC forms, bank statements, chats, emails and medical notes. Because the generator places each entity, the ground truth is exact by construction and the supply of labelled data is unlimited and free. This was my primary training data.
• Microsoft's Presidio sentence faker, for evaluation. Its sentence templates, span conventions and entity taxonomy are kept exactly as they are; only the values are regenerated with Faker's en_IN provider, because the bundled records are Czech and Italian names and useless for an Indian-locale test. Sentences are then concatenated into multi-PII documents — the gold carries no character offsets, only (text, type, cluster), so concatenation composes by simple union with nothing to realign.

The split matters. The training documents are mine end to end, which means the scorer and the generator share an author and therefore share every convention. The evaluation documents are somebody else's structure with my values, which is the only arrangement that can surface a convention I got wrong without me noticing.

One deliberate addition to the evaluation set: distractor sentences whose only entities were dropped by the type mapping — organisations, cities, dates. They read as entity-bearing and contribute no gold at all. That is the "looks like PII, is not PII" pressure that my training generator never applies. These were generated using the Presidio sentence faker.

Data distribution

The training data generator has 53 types — recipes, each generating unlimited examples — keyed like govt_id|kyc_form|d4|c1|x0|p6: government IDs, KYC styling, 4 entities, 1 name form, no distractors, 6 units of filler.

The grid is not a full cross product. That would be 1,260 types, most of them uninformative corners. So, it is a bit custom to ensure data generated is real-looking.

Reward Design

RL needs a reward for every attempt. If a human has to read the output, the loop is too slow; if another model judges it, the reward is itself an unreliable model. So the first requirement is narrow and non-negotiable: correctness has to be decidable by a deterministic program.

I decided on a non-binary reward scoring function. It is a few hundred lines of pure functions — no model, no network call, nothing that can be argued with — and it scores every answer from 0 to 1 as a weighted sum of five checks:

Recall outweighs precision because a missed Aadhaar number is a regulatory incident and an over-redacted street name is merely an annoyance.

Potential prevention of reward hacking Emit nothing and recall collapses; emit everything and precision collapses. RL will find a shortcut if one exists, so both have to be unprofitable before training starts. Grounding ensures that the model doesn't make up stuff.

The score is dense, not binary, and that makes it easier to find a useful gradient.

Finding the advantage - is there scope to learn more from RL ?

Next was to see if the base model actually had a scope to improve or already solves our problem. I used standard metrics to quantify this. The samples where we couldn't find a useful gradient is essentially not helpful data. As I mentioned earlier, this is essentially 'trial and error'.

Before going further, few definitions (aka jargon) used in the context of RL

• Environment — for a language model there is no game world or a setup. It is a prompt plus a scoring function: show a document, let the model answer, score the answer.
• Rollout — one complete attempt. Sampled at temperature 1.0, not greedy, because the whole method depends on the same prompt producing different answers on different attempts.
• Group — the k rollouts generated from one prompt. This is the unit that matters. Why groups of K ? That is because of the way GRPO works.
• Policy — the model being trained, understood as a probability distribution over next tokens. This is the policy that we optimise using the Grouped Relative Policy Optimization (GRPO).
• Effective fraction is the share of groups that are useful, and it is the one that says whether RL can learn here at all.
• pass@k asks whether any of the k attempts cleared the bar. This says whether model can give the right answer.
• Mean reward says how well on average was the task ever done properly.

So, to find data where where a useful learning signal exists, I need to setup an environment where I can generate rollouts, score them with a reward function, and compute relative advantage within each groups of K.

The scale of rollouts follows from the grid. Each of the 53 types gets 10 documents, and each document gets k rollouts:

53 types × 10 documents          =    530 prompts
530 prompts × 32 rollouts        = 16,960 rollouts

Two passes were run this way, at k=8 and k=32.

Running the untrained model across that grid shows where the difficulty actually lives:

Density dominates every other axis. Recall falls threefold from one entity to ten, and the chance of a clean answer in eight attempts goes from 97% to 11%. Precision meanwhile rises, from 0.498 to 0.575 (not shown in the graph above) — so the model is not getting noisier under load, it is getting incomplete. For PII redaction use case, that can be dangerous.

Why any of this is learnable

GRPO generates k answers to one prompt, scores them, and computes each answer's advantage as its distance from its own group's mean:

advantage_i = (reward_i − mean(group)) / std(group)

The group is the baseline here.

If all k answers score the same, the spread is zero, every advantage is zero, and that prompt teaches nothing. This type is essentially a useless sample

This is also where the dense reward earns its place. Take a group of eight failures. Under binary scoring: eight zeros, no spread, no advantage. Under partial credit: eight attempts that fail by different amounts, real spread, a usable advantage.

Whether a task is trainable is not purely a property of the model. It is a joint property of the model and the reward you wrote.

Interpretting the base model performance

The first run of the grid produced a mean reward of 0.303, and roughly a quarter of that shortfall had nothing to do with finding PII.

The model has to return JSON. When it returns something that is not JSON the score is 0.0 regardless of content — a gate failure — and it was failing that gate 23% of the time. A quarter of the apparent incompetence was punctuation.

The cheap fix is guided decoding, and it needs no training at all. Rather than sampling freely and hoping, the sampler is constrained at every step to tokens that keep the output matching a grammar. In vLLM that is a guided_decoding regex or JSON schema passed with the sampling parameters; it compiles to a finite-state machine over the vocabulary and masks every token that would break the pattern. Malformed output becomes structurally impossible, if we use this method.

Re-running the identical grid under the constraint:

+0.134 with zero training. That number is worth isolating, because it separates two problems that usually get argued about rather than measured. This is the cheapest reward lift I got in the entire exercise ! Formatting has exactly one correct target shape, which makes it a supervised fine-tuning job — and here it did not even need that much. The remaining 0.563 to the ceiling is capability, which I hope can come out of RL !

Guided decoding is also the better instrument than fixing format by SFT training, because it changes no weights and so cannot leak task knowledge into the model being measured. It is not free at run time — grammar-constrained sampling in vLLM ran at 5,648 tokens/second against 23,543 free-form, a 4.2× throughput penalty — which is fine for a measurement job.

Increasing K can potentially make more of your data useful

The k=8 pass reported 10 of the 53 types as never solved. Essentially their pass@8 scores were 0. The obvious reading is that these are beyond the model, RL has nothing to reinforce, and they should be dropped from training.

So I repeated the pass at k=32. Hypothesis was that maybe there are right answers lying around in the model output distribution, which we cannot capture at k=8.

Eight of the ten flipped to trainable.

Summary of the metric movement when we changed measurement from k=8 to k=32. The effective fraction increase tells you that we got more data than before which are trainable.

The RL scoping verdict: 51 of 53 types carried a usable advantage, with effective fraction at 98–100% almost everywhere, including types solved about 1% of the time.

Training runs

Read as a loop: at each training step, sample 4 prompts and generate 8 constrained rollouts per prompt, producing 32 trajectories. Score each trajectory with the reward function and use GRPO to calculate relative advantages among the eight responses to each prompt. Use those advantages to estimate the policy gradient, while applying a 0.02 KL penalty to discourage excessive deviation from the reference policy. Update the model starting with a learning rate of 2e-6, gradually decay it to zero, and repeat for 150 optimizer steps — yielding about 4,800 total rollouts.

Reward roughly doubled, and held on 11 document types that were never trained on (the held out set). Every reward component rose together — recall eightfold, alongside precision, grounding and coreference — which is the standard check for reward hacking, since a model gaming one term shows that term running away while the others stall.

Reading an RL training run is a bit different from SFT run

Train reward is not a learning curve. In SFT, training loss falls smoothly because every step sees the same kind of example. Here each step draws 4 prompts from types that differ enormously in difficulty, so most of the step-to-step movement is which types were drawn, not whether the model improved. Read it over ~20-step windows, or not at all.

The loss is not the metric. The SFT habit of watching train-loss against eval-loss maps here onto train reward against sealed-eval reward.

The degenerate-group rate is the stopping signal, and it has no SFT analogue at all. The share of groups where all eight rollouts scored identically and therefore produced no gradient. It is exactly 1 − effective fraction, measured live. It climbed from 0.03 to 0.37.

Final results

These numbers are the end of several iterations rather than one run, and the config moved as issues surfaced. Recall and precision started at 0.40/0.25 and were rebalanced to 0.325 each; weight later shifted to type accuracy (0.10 → 0.25) once the component log showed it was the only term still varying. Training settled at 150 steps, because external scores peak around there and decay afterwards.

Measured on 600 documents with 3,214 gold spans across 12 entity types, written by someone else. Strict token F1 requires the right span and the right label; typeless requires only the right span. A 0.5B model went from 0.105 to 0.406 on data it had never seen, for about a dollar of GPU.

Why external precision falls as training continues is the open question. It drops at every checkpoint — 0.391 at step 50 to 0.253 at step 300 — while the reward's own precision component sits near 0.96 and flat throughout, because a document made only of PII contains nothing that should not be extracted. Something about longer training makes the model reach further, and the training signal cannot see it happening. I do not have the answer yet !

Training a model further on your own examples is called fine-tuning, and the claim only means anything if you can measure it. So I picked three jobs where a computer — not a human reading the output and nodding — can decide whether an answer is right. I took these machine verifiable use cases to also try and learn how to do Reinforcement Learning with Verifiable Rewards (RLVR). I also thought that these are narrow enough for a small model to learn. The three use cases I tried are:

• Text to PromQL. A question in English goes in ("how much memory is this service using?"), a query in a monitoring language called PromQL (Prometheus Query Language) comes out. You can just run the query and see whether it returns what it should.
• Address cleanup. A messy Indian postal address goes in, tidy labelled fields come out. Since I generated the messy version from the tidy one, I know the right answer exactly.
• Screenshot to HTML. A picture of a web page goes in, the code to rebuild it comes out. Render that code and compare the result to the original picture.

The common shape: a program can score the answer. That is the constraint the whole project rests on, because it means I can measure a change instead of forming an impression of one.

What follows is a brief description of the broad learnings that I had.

The training data, and the part of it I could not see

Training examples came from two places. Where Hugging Face — a public library of datasets and models — already had something close to my task, I started there. Where it did not, I generated examples myself. I thought generating data would be pretty easy given how good LLMs are now-a-days. For addresses that meant starting from the real India Post directory of pincodes and localities, building a clean address, and then deliberately messing it up. That way the clean version is the right answer by construction, for free, and I never have to label anything by hand.

Then I looked at what I had actually produced, and found three problems. None of them showed up as an error. All of them capped how good the model could get.

I accidentally built a village dataset. I picked pincodes uniformly at random from the 19,238 that exist. India has far more villages than cities, so uniform picking gives you mostly villages: Mumbai got the same chance as any hamlet. Across 2,300 generated addresses the big metros turned up about eleven times, and Kochi and Gurugram never appeared at all. Real address traffic is the other way round — it piles up in cities.

One of my mess-up rules did nothing 95.7% of the time. People write city names badly, so one data generation rule replaced a proper name with a sloppy one — Bengaluru becomes Bangalore, or blr, or Banglore. Teaching the model to undo that was close to the whole point. But the rule worked off a hand-written table of ten cities, and if the address's city was not in the table it quietly left the address alone. My dataset was mostly villages. So the table missed almost every time, and the single behaviour I most wanted to teach was present in about 4% of my examples.

My table disagreed with my own directory. I had typed those city names from memory, and the official directory does not use the names I do. It says Ernakulam, not Kochi. Gurgaon, not Gurugram. There is no district called "Delhi" at all — it is nine separate sub-districts. Every name that did not match was a city that silently never appeared.

None of this is visible in the row count, and the row count is what you instinctively check. Four thousand examples sounds like plenty.

What matters is not how many examples you have. It is how many genuinely different situations they contain.

There is a second trap in generating your own data, and I walked into it. After training, the model scored 0.99 on my held-out test examples — near perfect. Then I typed in a real address I copied off a delivery app:

d1301, sattva sanctity, sarjapura road, bengaluru

It misspelled the building name and filed the city under locality. The model was not broken. My generator always wrote addresses in the same field order, used thirteen invented building names, and always included a locality. Real addresses skip fields, arrive in any order, and are full of proper nouns no generator would invent. My test examples came out of the same generator as my training examples, so all that 0.99 ever proved was that the model had learned my simulation of the problem. Fifty real addresses set aside at the start would have exposed this on day one.

Grading the answers, and then grading the grader

To measure anything I need a scorer: an ordinary program that takes the model's answer and the correct answer and returns a number between 0 and 1. No second model is involved in judging — it is just code. For PromQL it runs both queries and compares results. For addresses it compares the fields one by one. For screenshots it compares the rendered page against the original.

Then there is a step I nearly skipped, and it turned out to matter more than anything else in the project. Hand the scorer the correct answer, pretending the model produced it. It has to come back with exactly 1.0. A perfect answer must score perfectly. If it scores 0.87, the scorer itself is wrong, and every number the project produces afterwards is partly measuring that error rather than the model.

I ran this check — gold against gold — on every task, and it found a bug every time. On screenshot-to-HTML task, it found four. Two are worth explaining, because they are the kind of thing that hides forever:

It was comparing where things appear on screen with where they appear in the code. A block on a web page has two different orders: the order you see it in, and the order it is written in the HTML source. CSS can make these disagree — a sidebar can be written second in the source but displayed on the left. My scorer took the model's on-screen order and compared it against the original's source-code order. So a flawless reproduction of a page with a sidebar was marked down for the original page's own layout, scoring 0.429 out of 1.0.

Ties were broken by luck. To compare two pages I sort their blocks by position — down the page first, then across. But two blocks can sit at the same position, and then the sort has to fall back on something else to break the tie. It fell back on whatever order the blocks happened to be sitting in, which was different for the two lists being compared. The result: 117 documents came out "different" from themselves. The fix was to add a final tiebreaker so that identical inputs always sort identically.

The other two were worse in effect. Comparing two empty pages returned 0.55 instead of 1.0. And position was scored on exact pixels, so a layout shifted 40 pixels — indistinguishable to a human — scored 0.117, and at 120 pixels it scored zero. That last one meant a fifth of the available score was impossible for any model to earn, because no model reproduces pixel positions exactly. I would have paid to train against a target that could not be reached, and then concluded the model was incapable.

The address scorer had a bug with a nastier shape. One of my mess-up rules deleted the pincode from the input, but the expected answer still contained it. The only way to score well was to invent a six-digit number that was not there. I was one run away from explicitly training a model to make up postcodes.

Write the scorer first. Then try to break it.

Ask what the base model can already do

The most useful hour I spent was the one where I trained nothing at all. I ran the untouched model — Qwen3.5-0.8B, a small open model — against my scorer, and looked at the results field by field.

The address task has eight fields. Five are copied out of the input: premise (flat or plot number), building, street, locality, landmark. Three have to be produced in an official form: city, state, pincode. Alongside those I score grounded, which asks whether the model stuck to what the input actually said instead of inventing content.

Qwen3.5-0.8B, before any training (300 held-out addresses)

grounded  █████████████████████  1.00   already solved
pincode   █████████████████░░░░  0.82   already solved
landmark  █████████████░░░░░░░░  0.61
field_f1  █████████░░░░░░░░░░░░  0.44   how well fields match overall
geo       █████████░░░░░░░░░░░░  0.42   do city, state and pincode agree
city      █████░░░░░░░░░░░░░░░░  0.26   ← the actual problem

                     overall score  0.51

This changed what I built. The model was already refusing to invent things, and already pulling pincodes out of the text 82% of the time — effort spent there could win almost nothing. Turning a sloppy city name into its official form sat at 0.26, and the failures repeated: shown a city it did not recognise, it guessed a familiar big one.

Had I skipped this hour, I would have built a tidy dataset covering all eight fields evenly and spent most of my budget teaching the model things it could already do.

Training, and knowing when to stop

So the training set is not a fair sample of the task. It is deliberately skewed towards the places the model was failing — heavy on unfamiliar city names, light on the pincodes it already handled.

Then you train, and the question becomes when to stop. Two numbers move while training runs. Training loss measures how well the model fits the examples it is being shown; it will keep improving for as long as you let it run, which sounds good and is not — past a point it just means the model is memorising. Eval loss measures the same thing on examples it is not being trained on, which is what you actually care about. So you save the model periodically — each saved copy is a checkpoint — and keep the one where eval loss was lowest.

The textbook picture is a U: eval loss falls, reaches a bottom, then climbs again as the model starts memorising. My run did not do that. Eval loss dropped hard for the first hundred steps and then went flat — 0.4996, 0.4924, 0.4865, 0.4969, 0.4912, 0.4881, 0.4873 — jittering inside a band about a thousandth of a point wide, while training loss carried on down from 0.48 to 0.38. The memorising is real, but you see it as the gap opening between the two lines, not as the eval line turning upwards.

That makes "keep the best checkpoint" a less precise instruction than it sounds. Step 160 was picked as best at 0.4865; step 240 finished at 0.4873. The difference is 0.0008, which is far too small to mean anything. What the curve honestly told me was when to stop — past about step 120 I was spending money for nothing — not which exact checkpoint to ship.

Did it learn the job, or memorise my examples?

This is the question that makes or breaks the whole exercise, and answering it needs the test examples to be held back in the right way. Splitting rows at random is close to useless here, because a random test row was built from the same ingredients as the training rows. So I held out whole categories instead: six entire states appear nowhere in training, not in one single address, and two of the ways of messing up an address were never shown during training either.

Then I compared the trained and untrained model as two arms of the same run, over the same addresses, switching the trained part on and off — so nothing but the training differs.

                                          score
base model, untrained                      0.51
trained, familiar territory                0.63   +0.12
trained, mess-up types never seen before   0.64
trained, states never seen before          0.55

Read the last two lines against the first. The trained model handles kinds of mess it was never shown, and places it was never shown, better than the untrained model handled the easy case. It did not simply memorise the cities in its training data — it picked up a method: find the city in the text and rewrite it properly. Had I split rows at random, every one of these lines would have read the same and I would have learned nothing.

The same split also showed me what did not transfer. Broken down by field:

                  familiar states   unseen states
pincode                0.94             0.92      transfers fine
city                   0.46             0.34      degrades
geo consistency        0.69             0.47      falls apart

The pincode is sitting right there in the input, so reading it is a skill that works anywhere. Geographic consistency is different: to check that the city, state and pincode agree, the model has to know which state a pincode belongs to. That is not a skill, it is a fact — and facts about states I deliberately withheld were never learned. The model answers confidently and wrong.

Fine-tuning is good at teaching a method the model can apply to whatever is in front of it. It cannot install knowledge you kept from it.

The fix was not more training. Turning a pincode into a state is a lookup in a table I already had in code. It should never have been the model's job.

You run out of data before you run out of problems

Eventually the data stops coming. You have used the public datasets that were close enough to be useful, your generator is producing variations on things it has already produced, and the honest description of what happens next is that you stop looking rather than that you finish.

That is survivable. What is not survivable is trusting the number you happen to be holding at that moment.

The temptation at that point is to run the old model over one batch of examples, the new model over another, compare the two averages and call the difference an improvement. That does not work, because examples vary enormously among themselves — some are simply harder than others — so two averages can differ for reasons that have nothing to do with the models. On a small sample that difference can easily be larger than the effect you are looking for.

The fix costs nothing: run both models over the same examples and compare them one at a time. My eval script ends up doing something stricter still — it loads the model once and switches the trained part on and off between the two arms, so the two runs share not just the same examples but the same weights, the same batching and the same sampling. Nothing differs except the thing being measured.

Then you deploy, and real traffic starts finding cases your generator never imagined — a field order you did not think of, a building name no generator would invent, a question phrased in a way your training set never covered. This is not a failure of the method, it is the method: production is the only source of examples that is not your own simulation of the problem. The useful posture is to expect breakage, keep the machinery that turns each break into a new test case, and stop treating the held-out set as a finish line.

Would Reinforcement Learning have helped?

There are two broad ways to improve a model. Supervised fine-tuning — everything above — shows it the right answer and says "produce this." Reinforcement learning never shows it an answer: it lets the model try, scores the attempts, and pushes it towards whatever scored better. RL is the glamorous one, and I wanted it to work. I tried it on all three tasks. It never once helped, and eventually I understood why.

Reinforcement learning can only strengthen behaviour the model already produces some of the time. It shifts the odds between things the model already does. It cannot add something that was never there.

The algorithm I used, GRPO (DeepSeek fame), makes this very concrete. For one question it generates several answers, scores them all, and works out how much better or worse each was than the average of that group. That comparison is the entire learning signal — and if all the answers in a group score the same, there is nothing to compare, the signal is zero, and that question teaches the model nothing at all.

So before spending anything, ask the untrained model one question: generate eight answers to each prompt and sort every prompt into three piles.

never solved      all 8 attempts fail       → nothing to reinforce
always solved     all 8 attempts succeed    → nothing to improve
mixed             some succeed, some fail   → THE ONLY USEFUL PILE

That middle pile is the entire fuel supply. It can be empty for two opposite reasons: the task is too easy and the model always succeeds, in which case you need harder examples; or the task is too hard and it always fails, in which case you need supervised fine-tuning first, to get the model to the point where it sometimes succeeds.

There is a companion measurement called pass@k: let the model try k times and count the question as solved if any attempt worked. Comparing one attempt against eight tells you how much ability is in there but coming out unreliably.

Twenty-seven points of gap looks like a green light. It is not, and the difference is subtle but decisive: pass@8 counts a question as solved if any of eight attempts worked anywhere across the whole set. GRPO needs something stricter — a single question where some attempts succeed and others fail. Measured that way, on the address task:

8 attempts each, across 200 prompts

never solved     0.995      every attempt failed
always solved    0.000      every attempt succeeded
mixed            0.005      1 prompt in 200

One prompt in two hundred had anything to learn from. Dead on arrival — and I knew it for the price of one round of generating answers, rather than a training run. Fine-tuning first raised that to one in a hundred. Still nothing.

Text-to-PromQL is where I stopped arguing with the result, because there I ran RL anyway, twice, and kept every version:

                       valid queries    correct results
base model                  0.33             0.00
RL on the base model        0.33             0.00      ← nothing moved
after fine-tuning           1.00             0.22
RL after fine-tuning        1.00             0.22      ← nothing moved

Two runs, and both landed exactly where they started. From the base model there was no correct behaviour to strengthen. From the fine-tuned model the behaviour was already consistent, so every group of attempts scored alike and there was nothing to compare. The only thing that ever moved these numbers was showing the model the answer.

RL clearly does work — it is behind a great deal of recent progress. But the conditions it needs are specific: a task the model can genuinely already do and merely does unreliably, plus a scorer you trust completely. On small models and narrow, well-specified jobs, I did not find that situation. Supervised fine-tuning was sitting right there, and it worked.

What this was actually about

The hypothesis half held up. On addresses, a 0.8-billion-parameter model went from 0.51 to 0.63 and kept most of that on states it had never seen. On PromQL, fine-tuning took a model that produced valid queries a third of the time to one that produced them every time, and took correct results from zero to a fifth. Both for an hour on a rented GPU, at a running cost nowhere near a frontier API call. Screenshot-to-HTML I never got past the starting line, probably due to every model being very good at coding tasks these days. So there I have a measurement rather than a result.

But almost nothing I learned was about training. Training is a script and an hour of rented compute — a few hundred lines, and the least interesting part of the project.

Every expensive mistake was in the data or the measurement, and every one was silent. Nothing crashed. The curves looked healthy. The numbers looked plausible. A scorer that could not score a perfect answer as perfect. A dataset where the key lesson appeared in 4% of rows. A test set that could only ever measure my own simulation. Not one of them announced itself; each was found only on debugging a bad model output.

Which is the finding, and it is not a subtle one:

In fine-tuning, the datasets are the work. The training data decides what the model can learn, and the evaluation data decides whether you are entitled to believe it. Everything else is plumbing !

In this series of blog posts, I will jot down my learnings and thoughts from the various experiments I will be doing in the quest to understand GenAI deeply.

I have been playing around with LangChain this weekend. I asked my friend Bard to describe LangChain in about 50 words.

LangChain is a software development framework that simplifies the creation of applications using large language models. It provides a standard interface for LLMs, a selection of LLMs to choose from, and examples of end-to-end applications.

The one abstraction that stood out for me was Agents. Agents are essentially LLMs which are taught to think step-by-step and reason and act at each step. This prompting framework is ReAct framework. For a question, LLMs are prompted to think step-by-step and tries to reason about the next action to take. Agents are given access to a bunch of tools (e.g Google search, Calculator, Code Generator, etc) which lets them actually take those actions. The outputs of the actions taken are observed and the LLM reasons about the next action to take from those observations. This though-action-observation loop continues and it stops when there is no action to take and the question is answered.

To play around with this, I created a simple app based on LangChain agents which loads your CSV data and lets you chat with it. The agent used Pandas as the tool to do operations on the dataset. I used the Kaggle sale conversion optimization dataset for this experiment.

Here are some of the snippets of the chats I did with this agent. The verbose logs depicts how the agent arrives at the result.

Here, the agent was able to understand what a click through rate means. This is sort of expected since it is a pretty common term and the LLM would have an understanding for this.

The agent wasn’t able to understand what conversion rates meant. It instead did all the calculations based on conversions. An honest mistake I guess !

It is fascinating how the agent built up the Pandas query step by step. By this point, I am not at all surprised that the LLM understood that CTR and click through rate meant the same.

It seems prompting smartly makes an LLM smarter. Ideas like chain-of-thought reasoning forms the basis of ReAct. I also stumbled upon this — https://www.promptingguide.ai/ , which is a great guide into explaining some of these concepts with examples.

Compute Heavy Applications

Web applications which primarily deals with doing expensive computations with minimal or no I/O calls are classified as compute heavy. For example — given a user and their shopping cart, compute the total cart value, given a user and recommended products, score using an ML model and rank them, etc.

In such kind of applications, the most important physical resource are the CPU cores. So, how do you come up with the number of CPU cores required for your application ? Well, we need to look at the requirements.

Let’s say we need to build an e-commerce mobile app which needs to display a ranked list of recommended products to the user on its homepage. We are particularly interested in estimating capacity for the Ranking Service which takes a user and a set of products he recently interacted with as input and outputs a ranked list of products based on scoring and ranking the products using an ML model. Assuming that the ML model is stored in memory and there is no extra I/O required, this service is a compute heavy service. Let’s answer a few basic questions to understand the scale of the system.

What is the maximum no of concurrent users accessing the homepage?Around 100K users at peak.

What is the expected latency of a single homepage render for an optimal user experience?Latency should be 500 ms or better.

As per the requirement, 100K users will be concurrently opening the homepage. So, our application should be able to support 100K concurrency. To enable this, we need to have 100K application threads in our service. Here, a single application thread will independently handle a user at peak load. But how does application threads equate to physical cores?

4 physical cores handling 4 concurrent users. Here, concurrency == parallelism

In a compute heavy application, all the work happens in the CPU core. This means, all of what a single application thread does in our example, will happen in a CPU core. So, can we allocate 100K physical cores to our application? Of course we can ! Would it be optimal? Well, depends on the compute workload. Let’s answer a few more questions.

What is the on-CPU time for the ranking computation?Worst case, it can take around 300 ms of CPU time.

Since our compute takes at most 300 ms on-CPU time, we still have around 200 ms to spare as our target homepage render latencies are around 500 ms. So, if we give 100K physical cores, each core will handle a single user request and the response will be returned in 300 ms. Since we have an extra 200 ms at hand, can we use lesser number of physical cores by trading off latencies? Of course, we can! So, how many physical cores can we save? Here is where it gets tricky.

Concurrency != Parallelism

Often we confuse between concurrency and parallelism. In our Ranking Service example, if we had allocated 100K cores to support 100K concurrency, we would have used a parallelism of 100K, by virtue of 100K cores, to support 100K concurrency. While this is totally acceptable, it may not be very optimal.

To understand this better, let’s refresh our memory on how threads are scheduled on CPU. In a typical OS, threads are given a time-slice of the CPU in a round robin manner. Every time a thread is taken out of the CPU, the context (operating data, instructions, etc) is saved so that it can be re-used the next time it is scheduled on the CPU. This is called thread context switch. There is a cost associated with a context switch and it often depends on the context itself.

2 physical cores handling 4 concurrent users. . Here, concurrency != parallelism

Enough of OS theory ! So, how many cores should the Ranking Service use? The answer is — the number of cores at which off-CPU times (includes context switch times) of a thread aren’t exceeding 200 ms in a 500 ms window. That sounds complicated ! An easy way to determine the number is by performing a load test. We can perform a load test by iterating on the number of cores until your max latencies are 500 ms and concurrency supported is 100K. More often, this will be the limit of your system and CPU utilisation will be very high. Let’s say we performed the load test and the optimal number of cores came out to be 70K. So, we were able to support a concurrency of 100K using a parallelism of 70K.

Load Distribution

In our Ranking Service example, based on our load test, we can support a peak concurrency of 100K with 70K physical cores with a max (or 99th percentile) latency of 500 ms. What if the peak load happens only for 1 hour in a day and rest of the 23 hours, the peak never exceeds 50K? This means, for 23 hours in a day, most of our cores will be idle. This is never a good thing since we will paying for that compute unnecessarily.

If we know the times of the day when the peak load happens, we can spawn additional cores just before the peak hours and destroy the cores after that. This can even be done on demand when you detect extra load. These capabilities of auto / on-demand scaling are very common in pretty much every cloud service provider these days. This way, we can further decrease the overall physical cores footprint of our application.

Continuous streaming

The likes of Apache Storm and Apache Flink implements this approach. The idea is that the processing happens at the granularity of a single event in the stream. As soon as the event appears in the stream, the system reads it, processes it and moves it forward. Single event processing latencies are very low in this approach.

Micro-batches

This approach is born out of fitting a batch processing system into the streaming world. The key idea is to form batches of a small number of events and process them like a batch processing system would do. The system waits a certain duration to collect a batch of events before starting to process them.

Micro-batching approach is a natural evolution of a batch processing system. This extension was elegant and has a lot of positives going for it. But in certain scenarios, it starts to show weaknesses. For the remainder of the post, we will gloss over the good bit (because that is not interesting) and cover the bad bits (the interesting ones) fairly in detail.

The Good

High throughput

In most of the real life use cases like real time data processing systems, etc micro-batch based stream processing is good enough since we don’t really need the milli-second latencies. For real time data processing systems, we want higher throughput i.e maximum number of events passing through the system in a given time. But if we have a real time chat delivery, it might not be the best idea to implement it in Apache Spark.

Interoperability with batch processing code

With many minor changes, you could change your Spark stream processing workloads to batch processing and vice versa. All the key features available in the Spark batch processing world is also available in Spark streaming.

The Bad and the Ugly

While most of the drawbacks mentioned below are specifically based on my experience in Spark streaming but it should be equally true for any micro-batch based streaming system.

Starvation while consuming from multiple stream sources

Spark streaming allows you to write streaming applications which consumes from multiple streaming sources. For ex — a streaming application to generate real time recommendations can listen to order events and product-view events. While this can seem to be a good thing, it has a bunch of negatives which are hard to ignore. To understand this, let’s take a step back and understand how a Spark streaming application works when you have multiple streams involved.

If computation of the events in one of the streams is more expensive than the other, the throughput of the system with respect to the computationally cheaper stream suffers.

To workaround this, we will have to split this application into two, each consuming from one stream. This too has limitations.

• Splitting application based on stream sources can only be done if there are interactions between streams.
• Additional overhead of managing and maintaining two applications instead of one even though the business logic is very similar.

Plethora of jobs scheduled for one micro-batch

In spark, every action results in a job getting scheduled for the micro-batch. Suppose your streaming application does the following:

// pseudo code for generating popular products and user recommendations

// read events from stream sources  
product_view_events = sparkContext.readStream('product_view');  
order_events = sparkContext.readStream('product_view');

// group events for a user  
grouped_product_view_events =    product_view_events.map(_business_logic_function_).groupBy(user_id);

// group product view events for a product id  
pid_grouped_product_view_events =    product_view_events.map(_business_logic_function_).groupBy(product_id);  
pid_grouped_product_view_events.foreach(_update_view_count_per_day_);

grouped_order_events =    order_events.map(_business_logic_function_).groupBy(user_id);

// update recommendations for a user in a database  
grouped_product_view_events.foreach(_update_user_recommendations_);  
grouped_order_events.foreach(_update_user_recommendations_);

The above application will result in three jobs being scheduled, one for every action (calls to foreach function).

By default, Spark schedules these jobs sequentially. Which means, if the job corresponding to ‘popular products’ take a lot of time, user recommendations would be affected. Overall time taken for a single micro-batch to complete would be the sum of time taken by all the three jobs. Here, the total number of cores needed for the application will be the maximum required cores of all the jobs. But since job executions are sequential, some jobs which require lesser cores will bring down the utilisation of the total cores allocation. This isn’t very efficient.

There are ways to fix this, but it comes with its own challenges and drawbacks. Some of them are:

Running jobs within a micro-batch concurrently

There is a setting in spark which allows us to schedule and execute jobs within a micro-batch concurrently. Because of higher concurrency demand, we would need to allocate more cores to see the real benefit.

In our previous example, there are two jobs which originate from ‘product_view’ events. Running them concurrently would mean that there are multiple reads from the stream source and most of the compute is duplicated. In a sequential execution mode, we can eliminate this concern by caching intermediate results.

Breaking jobs into multiple applications

Breaking applications based on stream number of jobs adds an additional overhead of managing the code and deployment for each of these applications. This particularly becomes painful when the multiplexing factor between stream source and dependent jobs is high.

Similar to concurrent job executions, this also has the issue of compute duplication.

No HA during application upgrades

When you deal with real-time applications, you cannot compromise on its availability. In case of simple web service where you have many machines running the same application code, any deployment strategy supporting high availability would have some machine up and running at all times. This ensures a smooth upgrade of the application.

In a real time streaming application based on Spark streaming, this is not really possible out of the box. This is because there is one component — spark driver which is responsible for determining and schedule the next micro-batch. The driver is really a single point of failure. So, any downtime of driver causes a downtime of the application. But Spark master limits the damage by supervising it. So, any driver failure will be noticed by the Spark master and it restarts it.

Light at the end of the tunnel

Some of the drawbacks of enabling concurrent job executions like higher resource demand and inability to use caching is a necessary evil. If the re-compute cost is fairly small and you have enough capacity for the higher resource demand, then simply enabling concurrent job executions in Spark streaming can improve your application’s overall performance.

Continuous streaming approach doesn’t have a single driver responsible for cutting a micro-batch. This means that every node / thread in your distributed continuous streaming application will independently be consuming and processing events. This approach seems to eliminate some of the concerns we have with micro-batch based streaming. Fortunately, Apache Spark has an experimental continuous streaming mode, which do seem like a more suitable approach for real time streaming applications.

• Atomicity, if it has the ability to group multiple statements together and execute it as if it were executing a single atomic statement. There can only be two outcomes for such an execution, either all the statements execute successfully or all statements fail. This atomic execution is called a transaction.
• Consistency, if it can uphold your data invariants. This is highly dependent on the data modelling. If the modelling takes care of defining these invariants using the integrity constraint mechanisms that the database provides, a consistent (as per ACID) database ensures that those invariants hold true always.
• Isolation, if the database guarantees that concurrent transactions don’t affect each other.
• Durability, if the database ensures that every transaction that is committed are not lost. Absolute durability is not possible of course (total failure scenarios), but a durable database ensures that it safeguards against known patterns of failures.

In this post, we will pivot around Isolation. Absolute isolation is possible if all transactions are serializable i.e the final outcome will be the same as if all transactions were executed sequentially. The simplest way to achieve serializability is to force all transactions to execute on a single thread. This would, of course, incur a performance penalty and wouldn’t be suitable for real world applications. Before we look at better ways to achieve serializability, we will look at some of the weaker isolation levels that databases provide.

Why do we need isolation at all ?

Dirty reads. Ref: “Designing Data-Intensive Applications” by Martin Kleppmann

Here, user 1 sends an email to user 2. Sending an email involves adding an entry for the new email for user 2 and incrementing the unread count for user 2. If there were no isolation at all, there is a possibility that user 2 would see stale unread count even after receiving the new email.

Dirty writes. Ref: ‘Designing Data Intensive Systems’ by Martin Kleppman

This could happen because reads of un-committed records are possible. These reads are called dirty reads. Similar to dirty reads, there could be dirty writes as well, if a transaction overwrites an uncommitted value.

Read Committed Isolation

In a read-committed isolation level, dirty reads are prevented by making sure the database only allows reading committed entries.

No dirty reads. Ref: “Designing Data-Intensive Applications” by Martin Kleppmann

To prevent dirty writes, every transaction can take a row level lock which will prevent any other transaction to overwrite an uncommitted value.

Dirty reads can also be prevented by having the read transaction also taking the same lock but this could really hamper performance of read heavy systems. Any long running write transaction could block the read transactions for a long time. Ideally we don’t want writers to block readers or readers to block writers.

For every row, if we store both committed as well as uncommitted versions, any reader can just use the committed version while the writer continues its transaction by working on the uncommitted version. Once write transaction is complete, the previous committed version if replaced by the new one, seemingly atomically.

Although, there is a slight overhead of storing two version of every row, it pays off because of the significant improvement in performance.

Snapshot Isolation

Read skew in read-committed. Ref: “Designing Data-Intensive Applications” by Martin Kleppmann

This is a perfectly valid scenario in read-committed isolation. But from a user point of view, Alice sees a total of 900 in her bank accounts while she was expecting 1000. This expectation mismatch in Alice’s read query is called read skew or non-repeatable read. If Alice repeats her first read query, she will see 600 and a total of 1000 in her two bank accounts, which is consistent. The fact that Alice found the database to be in an inconsistent state needs to be fixed nevertheless.

We want the data to not change within a transaction i.e repeatable read. To achieve this, database provides a higher isolation level called Snapshot Isolation.

To achieve repeatable read, we have to ensure that read queries return the data which was last committed throughout the duration of a read transaction. From the read-committed implementation, we have the last committed data already which we can use here. But if we have a long running read transaction, we would need more than one version of committed data. This is because there could be multiple write transactions that could complete while the read transaction is in progress.

Multi-version Concurrency Control (MVCC)

Snapshot isolation is implemented using MVCC, where we have multiple versions of a row. Every new write, carries with it, the transaction id. For every read transaction, all reads will happen on the version of the data belonging to the latest transaction lower than the current read transaction id.

MVCC in action. Ref: “Designing Data-Intensive Applications” by Martin Kleppmann

Snapshot isolation did solve a lot of our issues but there are still many scenarios where it can lead to erroneous outcomes.

Concurrent read-modify-write operations tend to lose updates even in a snapshot-isolation mode. Ref: “Designing Data-Intensive Applications” by Martin Kleppmann

Here, the final counter value should be 44, but we lost one of the update. To achieve this, we need to make sure both these transactions happen serially. We will discuss Serializability in a future blog post.

References

• “Designing Data-Intensive Applications” by Martin Kleppmann

We will first look at how the data placement is done using Consistent Hashing and how this scheme drives the logical topology of the cluster. We will also see how cluster wide knowledge is made available to all nodes in the cluster in the absence of a centralised brain.

Consistent Hashing

Any get() or put() request for a key needs to be routed to the node which hold the partition which contains the key. But what is a partition exactly? A partition is a set of keys. As the dataset outgrows the capacity of a single machine, it is partitioned and spread across a number of machines. In this scheme, it is evident that for every request for a key, we need to be able to determine the partition and in-turn the node which holds the partition.

I will shamelessly plug in another post of mine describing the concept of consistent hashing. As it is shown in this post, consistent hashing requires the nodes to be laid out logically as a ring. In fact, it won’t be outrageous to think that the ring topology in Amazon Dynamo is more inspired by this than anything else.

Preference List

In the consistent hashing algorithm, once the point on the ring is determined by hashing the key, the next node in the clockwise direction in the ring is the node which holds the desired partition.

In a system where there is one master replica and a few followers for a partition, usually the master replica will be the one that is determined from the consistent hashing algorithm. The followers are chosen such that those nodes lie in a different physical machine, rack or even data centre.

In a master-less system like Amazon Dynamo, all the N replicas are determined using the Consistent Hashing algorithm. The next N nodes that lie in the clockwise direction in the ring will hold the replicas for the key range.

This list of N nodes found using the Consistent Hashing scheme are called the preference list.

Gossip Protocol

In the absence of a centralised brain, how are cluster wide knowledge, like cluster membership changes (node in or node out), etc notified to all nodes ?

Amazon Dynamo uses a protocol called gossip protocol to make information available across all nodes in the cluster. A node randomly chooses another peer node and propagates the information about the changes and this goes on until all the nodes are visited.

References

• https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

Quorum Failure

In Part 1, we saw how we can configure the values of R and W to achieve different levels of consistency in the system.

To achieve strong consistency, R and W values are configured such that R+W>N, where N is the number of the replicas for a given partition. For example, let N = 5, R=3 and W=3. In this case, if there is a network partition causing 3 of the nodes to be unreachable, the system becomes unavailable for both reads and writes for the keys in that partition.

In an eventual consistency configuration, a reasonable configuration would be N=5, R=1 and W=3. In this mode, R+W≤N and hence there could be replicas with stale values. Even in this mode, writes can fail if a network partition causes 3 of the nodes to be unreachable.

Didn’t we emphatically described Amazon Dynamo as an always writeable store ?

It sure is ! If majority of the replicas are unreachable, the system seeks help from the other live nodes. It uses a technique called hinted handoffs.

Hinted Handoffs

If nodes holding some of the replicas are unreachable, then Dynamo asks other live nodes to hold the data on behalf of the unreachable ones.

Let us suppose there are N replicas and 1 of the replicas are unreachable. In this case, when write arrives at one of the N-1 live nodes, it persists in its local disk and sends off the data to the other N-2 live replica nodes. It also sends off this data to 1 of the live nodes in the cluster, which is not the assigned replica for the partition. Along with the data, a context is also sent. This context contains information about the actual unreachable node. Since this node is not the assigned replica for the partition, it writes this in a separate location in its disk and periodically checks if the actual node is back online. Once that node is back online, it transfers the replicated data and deletes it from its local disk. This technique is called hinted handoff.

Hinted Replica Failure

In the above example, Node 3 takes the hint and persists in its local temp storage. By the time Node 2 comes back online, Node 3 fails permanently. In this scenario, Node 2 replica will go out of sync with other replicas (Node 0 and Node 1) permanently.

Hence hinted handoff is not sufficient to handle such permanent failures. We would need a mechanism to synchronise the replicas periodically. One way to do it would be to exchange a hash value of the entire replica and in case of a mismatch, synchronise it by sending all the data from one node to another. The node which has the maximum number of writes (although it is not clear from the paper) can be the one who initiates the data transfer in case of a hash mismatch.

Since this involves a large amount of data to be transferred over network, this is not desired. Another strategy would be to do these round trips for every key in the replica. This is clearly expensive and not desired. Both of these strategies require a linear scan on all the keys in the replica to determine the mismatches.

Amazon Dynamo synchronises replicas by storing the hashes of each record in a data structure known as Merkle Trees.

Replica Synchronisation using Merkle Trees

In Merkle Trees, each node contains the cryptographic hash of its children. Leaf nodes in the tree corresponds to the key:values pairs in the replica.

k4:v4 is a mismatch and it is found in log(n) time complexity

Using this data structure, mismatches can be found in logarithmic time complexity and hence it requires lesser data transfers between nodes.

Key set held by a node changes when the number of nodes in the cluster changes. This would require a re-calculation of Merkle Trees in those nodes.

References

• https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
• https://en.wikipedia.org/wiki/Merkle_tree

Ref: https://en.wikipedia.org/wiki/Amazon_DynamoDB

This will be a three part series of posts where each part will focus on a set of concepts used in the architecture of Amazon Dynamo.

• Part 1 will focus on the challenges of building an always writeable store. We will discuss the concepts to overcome those challenges.
• Part 2 will mainly focus on the failure handling techniques like hinted handoffs and replica synchronisation using Merkle Trees.
• Part 3 will deep dive into the ring topology in Amazon Dynamo architecture.

An always writeable store !

Writes are pristine in the world of an always writeable store. They should never go wrong. A typical e-commerce example that the paper mentions is of a shopping cart. Any cart edits done by the user should surely happen. The potential revenue impact of a failure in doing so can be huge in Amazon’s scale.

To understand what an always writeable store mean, let’s look at simple distributed data storage system where every record (key:value pair) is replicated across the cluster. This redundancy is mainly for fault tolerance as it lets us recover from a node failure by re-using one of the replicas. Redundancy also improves read scalability. We can scatter read requests for a key to all the replicas instead of overloading only one. One of the replicas is assigned master and other followers. Master replica is responsible for handling write requests and followers are updated either synchronously or asynchronously depending on the consistency mode we choose in the system.

In this architecture, let us examine if it is an ‘always writeable store’ by looking at some of the failure scenarios.

Master replica failure

Since there is a single node (the master) handling writes for a subset of records, this node becomes a single point of failure. Failure of this node can make our system unavailable to write requests. We can surely do better !

We can promote one of the follower replicas to be the new master and also bring up another follower replica to cover the loss. Is that enough ? Not quite! All the clients need to be notified of this change so that they update their local mappings (record to node mappings) and send all the future write requests to the new master instead of the old one which failed.

Till now, this system seems to be tolerant of failures towards write requests. What about the in-flight requests just moments before the master replica failure ? Those is inevitably fail and client would need to employ retries to and hope that before all the retries are exhausted, a new fully functional master comes up and client mappings are updated.

While such a system is fault tolerant towards node failures, there can still be some write request failures while the system is recovering. Retries could take a long time and can impact SLAs. This isn’t an always writeable system yet !

Eliminating The Master !

If there are no master replicas, who actually assumes the responsibility to handle write requests. The answer is — everyone !

This definitely invites more trouble. If multiple nodes handle writes, our data can go inconsistent pretty quickly. This might not be as bad as you think, if we are able to resolve the conflicts efficiently!

Either the server or the client can take the responsibility to resolve this conflict. Amazon Dynamo chooses to do this at client side. Clients will have the flexibility to have any strategy for conflict resolution. It could be as simple as ‘last write wins’ or a complex business logic.

Since every replica can handle writes, we will end up with multiple versions of the same record. We don’t necessarily need to keep around all versions. If we are able to establish causality relations between two versions of the same record, we can simply overwrite the ancestor with the child. The key to this is how we represent versions.

Vector Clocks

In a single machine, it is easy to establish causality relations if we use local time of the machine as the version. When we have multiple machines like in a typical distributed system, it becomes difficult to establish a causal relation between two events as local clocks might not be consistent with each other.

Vector clock is simply a vector of local clocks, where each of the machine in the cluster having its own local clock. Every source machine sends its own view of the vector clock at that point along with message to the destination machine. When the destination machine receives this message, it updates its vector clock using the following algorithm.

for each clock in destination_vector_clock:  
   destination_vector_clock[i] =   
          max(destination_vector_clock[i], source_vector_clock[i])

// local clock of the destination machine   
// corresponds to destination_vector_clock[k]

destination_vector_clock[k] = destination_vector_clock[k] + 1

Amazon Dynamo uses vector clocks to represent versions of every record. If there is a causal relationship between two updates, then the older version is overridden by the later one. Otherwise, both the versions are persisted.

Version change over time. Ref : https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

Quorum reads and writes

As we have seen above, to make the system highly available for writes, we need multiple nodes to participate in the write request. To offer sufficient redundancy, Amazon Dynamo allows us to configure the minimum number of nodes, W, that should accept a write request before it is deemed successful.

The same goes for read requests as well. Any read request is deemed successful only if at least R nodes return a valid response. If we set R+W>N where N is the total number of replicas, it mimics a quorum like system. This is because if R+W>N, then at least one node will be common between the read and the write. This gives us a strongly consistent system (CP).

R=2, W=2 and N=3

Tuneable consistency and durability semantics

Amazon Dynamo was built keeping in mind the need for flexibility. As per the CAP theorem, we can choose consistency or availability in the event of a network partition in a distributed system.

As we have seen in the above section we can make Amazon Dynamo behave as a strongly consistent system (CP) by configuring the values of R and W such that R+W>N.

Eventual Consistency

We can achieve eventual consistency if we keep a low value of R, say R=1 and R+W≤N. In this case, read request for a key will go to one of the N replicas but that replica could still hold a stale version of the key.

Durable writes

If W=1, writes will return as soon as it is persisted in one of the nodes. If that node goes down before replication is completed in at least one of the remaining replicas, we would lose that write. To make it durable, we should have W>1.

References

• https://en.wikipedia.org/wiki/CAP_theorem
• https://en.wikipedia.org/wiki/Amazon_DynamoDB
• https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
• https://en.wikipedia.org/wiki/Vector_clock
• https://en.wikipedia.org/wiki/Quorum_(distributed_computing)

Demand-filled Look Aside Caches

Web services which serves data from a database often use a cache on the side to prevent load on their DBs. Data in databases are usually stored in disk and serves as the sources of truth for the data.

As these databases are sources of truth, any modification to the data should be done directly in the DBs. There is escaping from writes. Write load needs to be handled by the DB itself. Can we do something about the read loads ?

Disk reads are typically slower than in-memory reads. This is because commodity disk hardware is made up of rotating disks, which involves position the head on the circular disk and then reading blocks of data. Solid-state drives offer random access but is way more expensive. RAMs, obviously, offer random-access and are way faster. Since RAMs are also expensive, we can cache a copy of the hot-data in it for faster access. Hot-data can be defined based on your use-case. Typically, we want to cache that subset of data which has the most probability of being read in the near future. A reasonable way to determine this subset is by simply caching recently read data.

Read requests in a web service can be made faster if we cache some subset of the data in RAM and always look into the RAM before querying the DB. This is called look-aside caching. What happens when the data changes ?

On any modification of the data, we can simply delete it from the cache. Any subsequent reads won’t find it in the RAM and fill fetch it from the DB. Since we want subsequent reads for this data to not go into the DB, we can fill the cache with this data to benefit in the near future. This type of a cache where the cache is filled on demand for making future reads faster, is called demand-filled look-aside caches.

Caching is powerful, but demands great responsibility !

Permanent in-consistency

When multiple concurrent clients observe a cache miss for a key and both of them try to fill the cache, the cached data could become permanently inconsistent with the source of truth database.

The above diagram illustrates a scenario where we could end up with a permanently inconsistent cache. This scenario is called stale sets.

Thundering Herd

The above diagram also highlights another problem. Client A and B, both observe a cache miss and try to fill the cache with the latest data from the DB. In a highly concurrent system, many such clients could read from the DB for the same key in a very short span of time. This can create hot-spotting for the key and can bring down our database.

Leases

To fix the stale-sets problem, we need to be able to discard the set operation initiated by Client A.

To do this, every time a client (A) observes cache miss for a key (k), our in-memory cache could be in one of these two states

• Client A is the first client to see this cache miss. In this case, the cache grants a lease to Client A. This lease tells A to go ahead and query the DB for the latest data and set it in the cache.
• Another client already has an active lease. In this case, the cache responds with ‘cache-miss’ but also telling the client to try after after some time.

This will make sure only one client goes to the database at a time. This will definitely solve the thundering herd problem.

But this still doesn’t solve the stale sets problem. In-memory cache should identify a set operation as stale or not-stale.

In line with our demand-filled model, we delete the key if it is updated in the DB. When we delete the key, we also expire the active lease, if any, for this key. Now, if we modify our set operation to carry the client’s lease along with the payload, our in-memory cache will be fully equipped to discard any set operation with an expired lease.

References

• https://research.fb.com/wp-content/uploads/2016/11/scaling-memcache-at-facebook.pdf
• https://instagram-engineering.com/thundering-herds-promises-82191c8af57d

Stateless Service backed by a DB

Every request to this service is re-formulated into a database query (SQL query) and these queries are executed remotely in the database. The service thread waits for the query to be completed in the database. Once the database is done processing the query, it responds back to the service with the ResultSet. This result set is then transformed to a response and sent back to the client.

Database queries can be expensive, especially, if it involves processing a large amount of data using operations like joins, unions, groupBy, etc. This can negatively impact the throughout and the latency of the system. One easy way to scale our service would be to cache data in-memory which are frequently accessed. This will improve the throughput of the system. For cache-misses, requests will still take a long time to complete. To improve latency, we might want to store all the data from the relational database into a low latency data store.

Storing Materialised Views

Every service will be interested in a subset of entities that exists in our database. Each of these entities could be dependent on multiple other entities. So, we need to make sure that when we cache the data, we cache a de-normalised view of the entity. This will enable us to reduce the number of calls made from the service to our low latency data store. This de-normalised view is also called a materialised view.

Change Data Capture (CDC)

Relational databases like MySQL, PostgresDB, etc has binary logging. Every row changes are logged in a file, commonly known as binlog. To materialise views, we would need to listen to these log events and process it in real time.

In the above design, we have a simple Bin Log Processor, which is responsible for doing two things

• One thread, say the I/O thread, which continuously polls for new events in the binlog. Every new event is handed over to the processing thread.
• Processing thread, materialises views and writes them to the sink. Every view in the sink corresponds to a row in the sink. Writes to sink are always atomic for a single row.

Please note that our Bin Log Processor is stateless.

Computing Materialised View

In the above example, the materialised view View is dependent only on one entity Entity_1. One entity object of View depends on one row of Entity_1 table. To compute View using our CDC design, we would need the I/O thread to hand over every row read/modify/update event to the processing thread. The processing thread then created the View in-memory and persist it in the sink. It can be reasoned that system will remain in consistent state in any kind of failure scenarios. Let’s look at some of the failure scenarios in detail to understand this better.

• I/O thread receives event but hand-over to processing thread failed. The sink will have a stale version (version 0) of our View entity. The view of the entity is delayed with respect to the DB but not wrong.
• Processing thread fails processing mid way. Since write to sink are always atomic for a single row, this failure can never leave the view entity in a wrong state. View version will be either 0 or 1.

Let’s complicate things a bit now !

In the above example, View update is complete only when both Entity_1 and Entity_2 are updated. A single transaction updates both the entities together.

Since our Bin Log Processor is stateless, single row event processing wouldn’t work in this case. We have to make Bin Log Processor transaction aware. We can do this by listening to two more events apart from the row events — START_TRANSACTION (TX_ID) event and COMMIT_TRANSACTION (TX_ID) event. When we encounter the former, we create a transaction event buffer for the TX_ID and start buffering all the events until we see a COMMIT_TRANSACTION (TX_ID) event. Once the processor sees this event, it can now start materialising the View entity.

The transaction event buffer is external to Bin Log Processor to keep the processing component stateless which helps in fault tolerance scenarios.

There are two primary components to this storage engine — hashTable and a log (append-only file). Let’s see how various operations work in this database

• write(key, data) : The data will be appended in the log at the latest offset, say, offset1. The first four bytes of the data will indicate the size of the data to be read. The key will be updated in the hashTable with the value in the hashTable being the offset to the data in the log.
• read(key): Offset to the data in log is found by doing a lookup in the hashTable. Once the offset is found, we seek to that offset in the file and start reading the data. Since the first four bytes indicate the size of the data, we would know when to stop reading.
• update(key, data) : Since we are using log, we will never go and update the data for an already existing key. If we were to do that, we will have to possibly re-write all data from the offset. So, we will just append this to the log just like the write operation. And update the offset in our in-memory hashTable. There would be old data lying around in the old offset, but let’s worry about that later.
• delete(key) : Delete is trickier. We will append a special record in the log, also called a tombstone in the log indicating that we need to delete data belonging to the specified offset. We will also remove the entry for the key from the hashTable.

So far, so good. All the operations run in constant time complexity. What about the data growth in our log ?

Log-structured Storage Segments

Data in our log is ever growing, since we are never really deleting anything. We will soon run into disk issues. One way to keep this data growth in check would be to have a background thread do the compaction of the log file. This will work but we cannot serve any requests while this is happening. This will compromise the availability of our system.

To make our system available for serving online requests, we can cut a snapshot of our current log file and create a new one instead of that by ignoring old and deleted data. These snapshots are also called segments While this is happening, another log file and hashTable can accept online requests. How would our write and read requests change in this approach ?

• write(key, data): Same as before except that we will append the data to the latest log file. The in-memory hashTable will have to store log file path along with the offset for a key as there can be multiple log files at a time in our system.
• read(key) : Determine the log file path and the offset for the given key and read the data in the same way we did in our previous approach.

To make sure this background cleanup operation happens more frequently we would need to limit our segment size. We can create these segments after a threshold size and do the compaction and merging in a background thread periodically.

This approach seems to have kept our data size on disk in check. The Achilles heal in our design is really the hashTable now.

• Fault-tolerance : In case of node failures, we will lose our hashTable. To fix this, we can persist our hashTable every time we cut a log segment. To recover from this fault, we can simply read the latest persisted hashTable and re-build our in-memory hashTable.
• Large number of keys: After a point our keys wouldn’t fit in memory. To maintain a hashTable on disk is possible but complicated and not very performant.
• Range queries : Scanning all keys between a given range will require us to look at all keys.

Sorted String Tables (SSTables)

Sorting is a powerful technique. Sorting all the keys allows us to reduce the size of our hashTable. Extending our previous idea of diving our file into multiple segments, we do the same here, only difference being that each of the file contains keys in a sorted manner.

Segment-0: ["aaabc":0, "aabc":980, "bbcd":1500]  
Segment-1: ["bcd":2000, "ccc";2200, "ccd":2500]  
Segment-2: ["cde":3000]

In this approach , we can limit the the in-memory hashTable to have only some of the keys. This is called a sparse-index. For example, sparse index will contains the following keys. “aaabc”:0, “bcd”:2000, “cde”:3000. Each of the keys in this index will be start of a segment file. This segment file is called SSTable. Any read operation for a key will require us to first determine the segment where the key would lie. This can be done by a binary search on the sparse-index. Once we know the segment, we simply read the segment sequentially from disk and return the data once we find the matching key.

We still need to scan through all the keys in a segment, which could make reads slow if segments are big enough. One possibility is to make the segments smaller. The con of this is that there would be far too many small segment files, which could be detrimental in our compaction and merging phase. If we make our segments larger, our sparse index becomes way smaller. In practise, we should be able to use more memory, if we remove the coupling of keys in the sparse-index to our segments. We can have our sparse-index have one key for every X KB of data. Scanning through few KBs of data is fast enough.

Keeping it Sorted !

The million dollar question is how do we keep the keys sorted on disk efficiently. To maintain keys sorted in-memory can be done using any of the balanced binary search tree implementations (AVL trees, RB trees, etc). All writes can be done in memory until a certain threshold of size is reached. After which, we can flush it onto disk. This in-memory data structure is called a memTable.

Keeping it sorted in memory is easy and flushing it periodically onto disk seems efficient. But how do we handle fault tolerance here. Any node failure can cause all un-flushed data to be lost forever. To prevent this, we can use a write-ahead-log (WAL). Any write operation, first appends the operation into the log and then written into memTable. The WAL writes are very fast as they are sequential writes. We can recover from any fault by building our memTable by scanning this WAL. We can limit the size of WALs by periodically rolling it since already flushed data needn’t be maintained in the WAL.

Every time a memTable is flushed onto disk, it creates a new segment file. Over time this can create many segment files. Again, borrowing ideas from our simplified storage engine, we can have a background thread to periodically do compaction and merging.

Log Structured Merge Trees (LSM Trees)

SSTables are periodically compacted and merged into larger SSTables. The idea is to keep a small number SSTable. We will see how small number of SSTables are key to having a good read performance. This cascading list if SSTables is the key idea of LSMTrees.

For each of the SSTable file, we have a sparse in-memory index. And each of these SSTable file can be ordered from oldest to latest. With this in mind, let us see how read and write operations work.

• write(key, data) : Writes the key:data into WAL and then writes it into the memTable.
• read(key): First look at the sparse-index for the memTable. If the key doesn’t exist, it could mean two things
• The key was updated/written long time back and could be present in an older SSTable file. In this case, we will have to go through all SSTables from latest to oldest. We return the data as soon as we find the key.
• The key is deleted. We can only say that a key is deleted after going through all the SSTables.

Performance of LSMTrees

As the writes involve one sequential write to WAL and an in-memory update to memTable, LSMTrees can give a very high write throughput.

For reads, we have to go through all the SSTables in the worst case. To get a high read throughput, it is important to keep a low number of SSTables. We can do this by doing a size based compaction. In this, newer SSTables are merged into older and larger SSTables.

For deleted keys, it still takes scanning all of SSTables to even determine if that key is deleted. One optimisation we can do is using a BloomFilter. BloomFilter is an in-memory data structure which approximates the contents of a set. It can tell you with 100% certainty if a key isn’t present in the set.

Range Queries

One advantage of storing keys in a sorted fashion is that it enables us to do range queries efficiently. For any range of keys [key_x, key_y], we can determine the exact SSTables to look for and hence limiting the key space to scan for.

Conclusions

This approach of storage and retrieval is used in several database systems like Google BigTable, Apache HBase, Apache Cassandra, etc.

References

• https://dataintensive.net/
• https://hbase.apache.org/book.html#arch.overview