The arc's last lever, and the only one that changes what the model is. It is also the most reached-for and the most regretted — usually by teams who wanted the model to know something and got a model that sounds like it knows it. Everything here is the move you already met — predict, score, nudge — pointed at data you own. Stop wherever you like — each depth is complete on its own, and the first one contains the decision most teams actually need.
Here is the ladder, and it is the most useful thing on this page.
The order is not fashion and it is not caution for its own sake. It is a ranking by reversibility and blast radius. A bad prompt is a bad string. A bad retrieval index is a bad index, sitting outside the model, replaceable on a Tuesday. A bad fine-tune is a new model — one whose regressions are not itemised anywhere, which has to be re-evaluated from scratch, and which you now have to re-derive every time a better base model is published.
And there is a second reason, which is the one this guide is really about. The three rungs do not do the same job. Prompting and retrieval are good at supplying knowledge. Fine-tuning is good at shaping behaviour. Confusing the two is the single most expensive mistake in this corner of the field, and it is worth taking apart properly.
The ratio argument, inherited with its evidence. Pretraining moved every dial in the model against trillions of tokens. You are arriving with a few thousand examples. Whatever your examples say about the world is a rounding error against what the weights already absorbed — but whatever your examples say about how to answer is a signal with almost no competition, because the model has only ever been shown a comparatively tiny amount of that. The same corpus that is far too small to install a fact is precisely the right size to install a habit.
Which is the mechanical statement of a rule dive four made in passing: capability comes from pretraining; behaviour comes from post-training. And fine-tuning is post-training. It is scoreboard two — supervised fine-tuning, the same loss on curated transcripts — pointed at your transcripts instead of a lab's. Nothing new is invented when you fine-tune. The corpus was swapped.
The picture is an upload: the documentation goes in, the model comes out knowing it. What you actually get is a model that has learned the register of your documentation — its vocabulary, its cadence, its confident house style — while still guessing at its contents, because a few thousand examples cannot outvote the pretraining corpus on any question of fact. It is also stale on the day you ship it, and re-teaching it means another training run.
Two mechanisms, each doing the thing it is good at. Into the weights: format, tone, schema obedience, the shape of a good answer — the things you can demonstrate a thousand times and that do not change weekly. Into the context: the facts, freshly fetched, with a citation and an update path that is a database write rather than a training run. This split is not a compromise between the two rungs. It is what each is for.
One clarification worth making early, because it removes a lot of mystique: fine-tuning is not a different technology from the one that made the model. It is the same loop, the same loss, the same optimiser, run for a very short time on a very small corpus, starting from weights that are already good. Everything that is true about training is true about fine-tuning — including the thing dive four warned about, which Level 3 will make you watch happen.
The ladder is prompt → retrieve → tune, in that order, and most teams should stop before the last rung. Reach for it when the problem is behaviour — format, tone, the shape of the answer. Reach for the rungs above it when the problem is knowledge.
Full fine-tuning, honestly. Take the finished weights, put them back in dive four's loop, and feed it your data. Every dial is free to move. The gradient is computed for all of them, the optimiser keeps its own running state for all of them — typically several times the size of the weights themselves — and what comes out the other end is a complete, independent copy of the model. That is the honest description, and every cost in it follows from that one sentence:
Then somebody noticed something about the shape of the update. When you fine-tune a model for a behaviour, the total change to any given weight matrix — the difference between the tuned table and the frozen one — turns out to be low-rank: it can be reconstructed almost perfectly from a handful of directions. Which, stated in English, is exactly the intuition from Level 1. You are not rewriting the library. You are adding a reading style, and a reading style is a small object.
LoRA — low-rank adaptation, Hu and colleagues, 2021 — takes that observation and makes it the architecture. Dive three's learned tables stay exactly where they are and are marked read-only. Beside each one you put two skinny matrices: an A of shape d × r and a B of shape r × d, where the rank r is tiny — 8, 16, 64, against a d in the thousands. Their product is the same shape as the frozen table, and it is added to that table's output.
That last word is the whole design, and you have met it twice already. The residual stream adds; it never overwrites. LoRA does the same thing one level up: the original computation is untouched and a small correction is laid on top of it. Nothing is written back into the base model at any point during training. That is not an implementation detail — it is the reason everything below is true.
Now the census, in the tradition of this series. One square table, d × d. Move the rank and watch the arithmetic — the strips' area is the parameter count:
The headline, at a frontier-ish width of d = 8,192 and a routine rank of 16: 262,144 trainable dials against 67,108,864 frozen ones — 0.39%. And the ratio is 2r/d and nothing else, which has a consequence worth pausing on: the wider the model, the cheaper the same rank gets. LoRA is not a compromise that gets worse at scale. It is a compromise that gets better at it.
Four consequences, one sentence each, all of them descending from "the base is frozen and the delta is added beside it":
There is a further squeeze on the same idea: hold the frozen base in a quantised form — four bits per weight rather than sixteen — while the small adapter trains in higher precision on top. The base is read-only for the whole run, so the loss of precision costs you less than it would in a full fine-tune, and the memory it saves is the memory that decides whether the job fits on one accelerator. That is QLoRA (Dettmers and colleagues, 2023), and it is a large part of why single-GPU fine-tuning of very large models stopped being a research claim.
// the whole of LoRA. the base model is read-only for the entire run. y = x @ W // W is d x d and FROZEN — it never receives a gradient delta = x @ A @ B // A is d x r, B is r x d — the only dials that move out = y + (alpha / r) * delta // ADDED beside the table, never written back // at d = 8192, r = 16: // frozen = d*d = 67,108,864 dials you are not allowed to touch // trainable = 2*d*r = 262,144 dials you are // share = 2r/d = 0.39% <- the entire economics, in one ratio // unload A and B and the model is byte-identical to the one you started with.
Two things in that snippet are worth reading slowly. Line 3 is a normal forward pass and it is unchanged — the tuned model still does everything the base model did, by the same route. And the scaling factor on line 5 exists because the adapter starts at zero: B is initialised to zeros, so delta is exactly nothing on the first step, and the tuned model is provably identical to the base model before training begins. Every difference you end up with is a difference you can point at.
For behavioural work — format, tone, schema obedience, task shape — a well-chosen rank matches full fine-tuning remarkably often, and the reason is the observation the method is built on: the update was low-rank anyway, so giving it more capacity buys little. Where full fine-tuning genuinely pulls ahead is where the change is large — a new language, a genuinely different modality of task, a substantial shift in the whole distribution.
The thing LoRA cannot do is the thing fine-tuning itself cannot do well: bulk-load knowledge. That limit is not about rank, and turning the rank up will not move it — it is the ratio argument from Level 1, and it applies to every rung of this ladder equally. Choosing full fine-tuning to fix it is paying an order of magnitude more for the same disappointment.
The mechanism, in one clause. The gradient answers exactly one question — which direction would have made this batch slightly less wrong — and it answers it for every dial it is allowed to move, including the dials that were holding something else. If a dial's current setting helps your corpus, it moves. Whatever it was doing before is collateral, and nobody is keeping score of it. There is no term in the loss for "and also stay good at everything else". There never was one.
This is catastrophic forgetting, it long predates transformers, and it is not a bug in anyone's implementation. It is what the move does.
Reading that is one thing. Watching it is another. Below is the training guide's lab, resurrected: a character-level model of 32,768 dials, pre-trained in your browser on the same prose corpus dive four used, then fine-tuned on a second corpus in a different register. Both curves are full evaluations, computed live. The original corpus is never trained on — its line is a pure measurement of what the fine-tune cost you.
Three things in there are worth doing deliberately. Run it once with replay off and watch the coral line climb while the violet one falls: that gap is the trade, drawn to scale, and nothing was added to the code to produce it. Then sample both registers before you reset — the prose sample comes back wearing brackets and underscores, which is what "the model got worse at English" looks like when English is all it had. Then switch replay on and run it again against the dashed ghost of the first run: the damage flattens, visibly, and the new-corpus curve takes noticeably longer to come down. Both halves of that sentence are the point. The mitigation is real and it is not free.
The mitigation playbook, which is short, because there are only so many places to intervene:
Now the cost centre nobody budgets for: the data. Every account of fine-tuning spends its words on the training run, which is the cheap part and the part that finishes overnight. The expensive part is assembling a few hundred to a few thousand examples that demonstrate exactly the behaviour you want, consistently, with no contradictions between them — and it is expensive because of where the examples come from.
They come from your product. Which means they are full of personally identifiable information that has to come out; mistakes your support team made on a bad day, which the model will learn as faithfully as it learns the good answers; inconsistencies between two agents who formatted things differently, which teach the model that either is fine; and duplicates that quietly reweight the corpus toward whatever you happened to log most. Curation is not preparation for the project. Curation is the project — and the well-worn finding holds: a thousand carefully curated examples routinely beat fifty thousand scraped ones, because every bad example is a gradient step in the wrong direction and the model cannot tell which is which.
And the discipline that decides the whole question. A fine-tune produces two numbers, and almost everyone measures only the first one:
// the harness that decides whether the rung was worth climbing. // eight lines. most teams write the middle one and none of the others. before_yours = eval(base, your_tasks) // the win you are buying before_general = eval(base, general_suite) // the tax you are paying tuned = finetune(base, your_data) after_yours = eval(tuned, your_tasks) after_general = eval(tuned, general_suite) report(win = after_yours - before_yours, tax = before_general - after_general) // if you cannot measure the win, you have not earned the rung.
You need a held-out set for your task — the win — and a general suite you did not tune on — the tax. Without the first you cannot tell improvement from wishful thinking, because you will be reading the outputs yourself and you will be inclined to like them. Without the second, the regression ships silently: the model is better at the thing you were staring at and worse at four things you were not, and nobody finds out until a customer does. If you cannot measure the win, you have not earned the rung. And the craft of that harness — what to measure, how to score it, when a delta is real rather than noise — is a dive of its own.
Fine-tuning always charges a tax, because the gradient has no term for "stay good at everything else". Replay mixes and small deltas limit it; only a before-and-after eval prices it. And the real cost of the project is the data curation, not the training run.
Format and schema obedience, at volume. The strongest and least glamorous case. If your product needs the model to emit the same structure on every call, you can either pay for a thousand tokens of instructions and examples on every single request, forever, or you can demonstrate the structure a thousand times, once, and stop paying. The arithmetic favours tuning as soon as your request volume is meaningful — and the reliability usually improves too, because a demonstrated habit holds better than an instruction competing with everything else in a long prompt.
Tone and register. Style is behaviour by definition, which makes it the thing the move teaches best. Prompts can describe a voice; a few hundred examples are the voice, and the difference shows up most in the cases you did not think to describe — the awkward refusal, the half-answerable question, the edge where a prompt's adjectives run out.
A tuned small model replacing a prompted large one. The strongest business case in the list, and the one with real money attached: take the task a large model does well behind an elaborate prompt, and teach a much smaller model to do that one task. Latency drops, unit cost drops by a lot, and on a narrow enough task quality often holds. The engine for it is distillation — use the large model to generate and grade the training data, then tune the small one on what survived. That is dive four's synthetic-data refinery, pointed at deployment instead of at the next frontier model: generate in bulk, keep only what passes inspection, train on the residue.
Domain vocabulary and interfaces. A useful subtlety, because this one looks like knowledge and is not. Teaching the model the shape of your domain — that these are the entity types, this is what a well-formed query against them looks like, these are the tools and this is when each is appropriate — is behaviour. Teaching it the contents of your domain is knowledge, and belongs one rung up. The same fine-tune can make a model fluent in your interfaces while it remains, correctly, ignorant of your data.
On-device and open-weight deployments. Where prompting alone cannot carry the product — a small model running locally, with no budget for a long system prompt and no larger model to fall back on — tuning stops being an optimisation and becomes the thing that makes the product possible. This is the open-weights consequence in practice: you can only tune what you can hold.
And the honest losses, which are the same list read backwards. Knowledge — go up the ladder to retrieval, every time; the weights are the worst place you could put a fact. Anything that changes weekly — a training run is a slow write to a slow store, and your prices, policies and inventory are not that. Tiny example counts — a few dozen examples is a prompt with extra steps and worse ergonomics; put them in the context and keep your evenings. And the treadmill, which the abyss prices properly: the base model you tuned will be superseded, and your adapter does not come with you.
One more case that deserves naming because it is the most common wrong reason of all: "the prompt is too long and unwieldy" is a legitimate motivation, and "we tried prompting and it didn't work" usually is not — not until someone has established why it did not work. If the model lacked a fact, tuning will not supply it. If the model was reading a contradictory instruction buried in paragraph nine, tuning will bake the contradiction in.
It will make it sound like it knows your product. That is not a lesser version of the thing you wanted — it is a worse failure than the one you started with. The gap you were trying to close was a model that answered vaguely, in the wrong voice, and was therefore easy to distrust. What you get instead is a model that answers in your house style, with your vocabulary, at your level of confidence, and is still guessing. You have removed every signal a user had for telling a real answer from a fabricated one.
The mechanism is Level 1's ratio and nothing more exotic. A few thousand examples cannot outvote a pretraining corpus on a matter of fact, but they are more than enough to install a manner. So the manner is what you get.
Knowledge wants retrieval — where the fact arrives at query time, with a citation, and can be corrected with a database write. The ladder exists because each rung down is harder to reverse, and this is the rung where an unreversed mistake gets served to your users wearing your own tone of voice.
Everything above tunes on demonstrations — here is a good answer, imitate it. There is a second scoreboard available to you, and it is dive four's preference machinery, sized down: instead of writing the good answer, rank pairs of answers and train the model to prefer the winner. Direct preference optimisation makes this practical without a separate reward model, which is why it is the version that reaches small teams.
The appeal is obvious once you look at what a product already logs. You have pairs everywhere: the draft that was sent versus the draft that was edited, the suggestion accepted versus the one dismissed, the answer that ended the conversation versus the one that produced a follow-up. Those are real preference signals from real users, and they are free.
The risks come down with the machinery, and they do not shrink proportionally. A lab's raters are trained, briefed, disagreed-with and audited. Your proxy metric is a thumbs-up button, and the model will optimise it — not the thing you hoped it stood for. Sycophancy is the canonical example and it arrives faster at small scale than large: people click approval on answers that agree with them, so a model tuned on clicks learns agreement. Reward hacking is not an exotic failure here. It is the default outcome of optimising a proxy you did not interrogate.
The honest version of this pod: preference tuning on product signals is powerful and genuinely underused, and it demands the evaluation discipline of Level 3 more than demonstration tuning does, because the thing being optimised is one step further from the thing you actually want.
Take two models fine-tuned from the same base, average their weights element by element, and use the result. This should not work. It works, often, and it has since been demonstrated enough times to stop being a curiosity: the model soup result (Wortsman and colleagues, 2022) showed that averaging the weights of models fine-tuned with different hyperparameters can beat the best individual member, at no extra inference cost — unlike an ensemble, which pays for every member at every request.
The intuition, which is a hand-wave and should be read as one: fine-tunes from a shared base stay in the same neighbourhood of the loss landscape, and within that neighbourhood the behaviour deltas compose roughly linearly. Average two such deltas and you land somewhere between the two behaviours rather than somewhere useless. LoRA makes this especially tidy, because the delta is not a metaphor — it is a specific pair of small matrices you can literally add up.
The limits are exactly where you would expect once you know that intuition. Models from different bases do not merge — there is no shared neighbourhood, and the average of two unrelated coordinate systems is noise. Adapters trained for conflicting behaviours average into something that does neither confidently. And a merge is not measured by anyone until you measure it: it produces a model nobody trained, whose regressions are nobody's fault, which makes Level 3's before-and-after harness non-optional rather than good practice.
Here is the cost nobody puts in the business case. Your adapter is tied to the base model it was trained against. A new base is published, it is better than yours-plus-adapter at most things, and your months of curated data now buy you a decision rather than an asset: re-run the tuning against the new base, re-evaluate everything, and re-qualify the result — or stay on a base that is falling behind.
Which means the honest way to price a fine-tune is not as a one-off project cost. It is a depreciation schedule: the artefact loses value on a cadence you do not control, and the maintenance is a standing commitment. Data curation survives a base change and is the durable asset; the adapter is the perishable one. Teams who keep their datasets under version control with the eval suite beside them re-tune in an afternoon. Teams who treated the adapter as the deliverable start over.
Which makes waiting a real engineering option and not laziness. If the behaviour you want is on the trajectory of general capability — better instruction-following, better structured output, better long-context reliability — the next base may hand it to you for the price of a version bump. Tune the things that are specific to you, because nobody else will ever build them; be slow to tune the things that look like everybody's problem, because everybody is working on them.
A short pod with a large blast radius, and it starts with a correction: "open weights" is not a licence, it is a distribution method. What you may actually do with the file is decided by whatever document came with it, and those documents differ from one another far more than the phrase suggests. As of mid-2026, releases described as open sit across at least three genuinely different regimes: permissive licences in the ordinary open-source tradition; use-restricted licences that grant broad rights but carve out named categories of use, or gate commercial use above a size threshold; and research-only or evaluation licences that do not permit production use at all.
Two consequences that catch people out. First, your adapter inherits the base's terms — it is a derivative of the model it was trained against, however small the file is, and no amount of "we only trained 0.4% of it" changes what the base licence permits. Second, the training data has its own separate licence story, and a permissive model licence says nothing at all about whether you were allowed to train on what you trained on.
The operational advice is unglamorous and it is the only advice that survives the release cadence: read the actual licence, for the actual version, before the work starts — not the blog post, not the model card summary, and not the phrase "open" in a headline. This is the one section of this guide most likely to be out of date by the time you need it, which is itself the point.
The arc asked one question three ways: when the model is not enough, which lever do you pull? Here they are together, in the order that has been implicit all along and is now explicit.
Reach for the context first. Change what the model reads by writing better text — free, instant, reversible, and it fixes more than anyone expects it to. Reach for the index second — retrieval, when the problem is knowledge the model does not have and should not be expected to; and its sibling, tools, when the problem is an action the model cannot take by writing. Both of those leave the weights alone. Reach for the weights last, when the problem is genuinely behavioural, when you can demonstrate the behaviour a thousand times, and when you have a harness that will tell you what it cost.
Each rung down costs more, reverses worse, and drifts further from what the machine is naturally good at. That ordering is the single most portable thing in this arc, and it will outlive every specific technique named in it.
And it is worth seeing where the descent ended up. The series opened with a claim that sounded reductive: an LLM is a next-token predictor in a loop. Eight dives later, that claim has not been amended once — it has only been unfolded. The chunks it reads and the geometry they live in; the attention that lets a token see its context; the one move, repeated at absurd scale, that put the numbers there; the two phases that explain every line of the invoice; and then the three levers you now hold — reading, acting, and being.
Which was the point of going down rather than across. Nothing in this stack is magic, nothing in it requires a new intuition every six months, and you can now argue about any of it from first principles — including the parts that had not been invented when this was written. That is what the descent was for.