Three dives have toured a machine whose every number was "learned" — a word doing an enormous amount of unexamined work. This one unpacks it: the single move that sets every dial, the four scoreboards it gets played against, and why the last three stages are a costume fitting on the first. (If "next-token predictor" is not second nature yet, start at the hub.) Stop wherever you like — each depth is complete on its own.
The last dive counted the dials: billions of numbers, sitting in learned tables, doing all the work. The dive before it admitted where those numbers start — as random noise, including the embedding table where meaning eventually lives. Training is the process that turns the noise into the numbers, and it has exactly one move:
Then do it again, with the next piece of text. Trillions of times.
The direction-finding step in the third line has a name — the gradient — and it deserves one sentence of demystification, because it is where people imagine magic. It is not search and it is not trial and error. Nothing tries a dial, checks whether the score improved, and puts it back. Calculus hands you the direction for all of the billions of dials in a single sweep backwards through the network — one pass, every direction, at a cost comparable to the forward pass that produced the guess. That is the only reason any of this is affordable, and it is the whole of the mathematics you need to hold.
Engineers wrote the grammar in, loaded the facts, coded the persona, added rules for what to refuse. Nobody wrote anything in. There is no line of code in a language model that knows what a noun is, no table of facts to update, and — as the hub established — no place where knowledge is stored as knowledge, so there would be nowhere to write it even if someone wanted to.
Engineers build the racetrack: the architecture (dive three), the data (this dive), and the scoreboard. Then the move runs trillions of times, and whatever configuration of dials scores better survives to be nudged again. The driver is not designed; the driver condenses. Dive two said this about the geometry of meaning — prediction pressure put it there. This dive is that sentence, industrialised.
And now the map of everything below. Modern models are built in stages, and the stages are not four different kinds of learning. They are the same move, four times, against four different scoreboards:
Same three operations every time. Only the source of the score changes — and each scoreboard leaves its own layer on the same pile of numbers.
One honest sentence about scale, because the rest of this guide leans on it. The move runs on thousands of GPUs for months. The first scoreboard consumes trillions of tokens. The other three, combined, consume a rounding error of that — and that ratio, not any mystical property of reinforcement learning, is why the last three stages cannot teach the model much it does not already know.
Nobody writes knowledge into a language model. One move — predict, score, nudge — runs trillions of times against four scoreboards, and everything the model is falls out of what each scoreboard rewarded.
The score, concretely. The model emits a probability for every token in the vocabulary — the hub's distribution, the same one you can watch in its sandbox. The scoreboard asks exactly one question of it: how much probability did you put on the token that actually came next?
Put 90% on the truth and you were barely surprised; the score is near zero. Put 0.1% on it and you were astonished; the score is large. That is the whole of the loss function — a measure of surprise, averaged over every position in the batch. (Its formal name is cross-entropy, and "average surprise" is not a simplification of it, it is what it measures. You will not need the formula anywhere in this guide.)
Two things about that make training practical rather than merely possible. The model is scored at every position at once — feed it a thousand-token document and you get a thousand predictions and a thousand scores from a single sweep. That is dive three's causal mask wearing its training hat: the mask hides the future from each position, so all positions can be computed in parallel without any of them cheating. And because the answer is simply "the next token", the labels are free. Nobody annotated the internet. The text labels itself.
Blame assignment — and the residual stream pays its IOU. Scoring is easy; the hard part is the third operation. "Which direction should each of seventy billion dials move?" is answered by pushing the mismatch backwards through the network, layer by layer, each layer handing the layers below it their share of the blame. That procedure is called backpropagation, it is named here once, and it is mechanically the chain rule from school calculus applied at scale.
Dive three left a promissory note about this: it said the residual stream — the bus, where every sub-block adds its output rather than replacing what was there — is what lets a blame signal survive the trip across eighty layers. Here is why. A layer that replaces its input forces the blame to pass through it and be reshaped by it; do that eighty times and the signal reaching layer one has been multiplied by eighty different things and is, in practice, either vanishingly small or wildly large. Addition gives the blame a direct route home: it flows to the layers below both through the block and straight down the bus, unaltered. Deep stacks are trainable because of an addition. That is the payoff, and it needs no calculus to see.
// the generation loop, run the other way. same machine, opposite direction. for batch in corpus: probs = model(batch) // ONE forward pass — a distribution at EVERY position loss = surprise(probs, actual_next) // how much probability landed on the token that came next? grads = blame(loss) // backwards through the bus — every dial gets a direction dials -= tiny_step * grads // the move: nudge all seventy billion of them, a hair // that loop is the entire secret. everything else is deciding what goes // into corpus and what counts as surprise.
Compare that to the hub's generation loop and the symmetry is exact: generation runs the model forward and keeps the tokens; training runs it forward, keeps the surprise, and sends it backwards. Same machine, same forward pass, opposite direction of travel.
Ten lines are one thing. Below, it runs — genuinely, in this tab, on a model small enough to fit in a page. Press Train and watch the score fall.
Four things in there are worth watching deliberately, because between them they are the level.
The line is the phenomenon. Not a metaphor for training — the actual score, measured on the last two hundred and fifty examples, plotted as it happens. It starts at 4.81 bits because a model whose dials are all zero considers all 28 symbols equally likely, and log₂28 is the surprise of guessing uniformly. It falls past the second dashed line — what you would score knowing nothing but how often each letter appears — and keeps going, because knowing the previous two characters is worth real money. A frontier lab watches exactly this line, on a chart with a much bigger x-axis, for months.
Nobody taught it English. There is no rule in there about q being followed by u, no list of words, no notion that a space separates things. Read the samples in order: noise, then noise with plausible spacing, then letter clusters that look like words from a distance, then actual words, badly spelled. Every one of those was bought by nudging 21,952 numbers downhill.
And watch a sample dissolve. Every so often a run of characters comes out as pure noise for half a word before recovering. That is not a bug and it is not the sampler misfiring: three kilobytes of English only ever contains a few hundred of the 784 possible two-character contexts, so most rows of that table were never scored even once and still hold the zeros they started with. Land on one and the model emits a uniform guess, because that is genuinely all it has. Readers of dive two's glitch-token pod have met this exact failure at production scale: an untrained region of a learned table decodes to nonsense, and no amount of training elsewhere fixes it.
And it stops improving. Twenty-two thousand dials and three kilobytes of text buy you word-shaped babble and nothing more. The model has no room for anything larger than a two-character habit. Scale is not a detail of this story; scale is the story, which is what Level 3 is about.
Now the corpus — because "the internet" is not a dataset. Between a web crawl and a training run sits a pipeline that is mostly engineering judgement, and every decision in it becomes a permanent property of the model:
Two facts about that last step are worth carrying around. The first: mixture decisions are product decisions. The same architecture, the same parameter count and the same total token budget, trained on two different mixes, produce two models with noticeably different characters — and the industry's working belief (stated as a belief, because it is not a law) is that a heavier code fraction improves reasoning-adjacent behaviour well outside code, presumably because code is text where long-range logical structure is unusually explicit.
The second closes a loop from dive two. The tokenizer was trained on a corpus assembled by these same choices — an English-heavy pipeline produces an English-heavy merge list, which is precisely why everyone else's token bill is higher. The multilingual inequity you can measure at the API is a data-mixture decision, wearing a different hat, two years later.
Scale anchor, for a sense of the pile: modern frontier corpora run to around 15 trillion tokens — Llama 3's published figure, one of the few public numbers of its kind.
And what comes out is not an assistant. It is a base model: the hub's alien. Erudite, capable, and utterly unhelpful — a pure continuer of documents that will answer your question with more questions if that is what the training text suggests comes next. Everything it will ever know it now knows. Everything it will ever do for you is still three scoreboards away. (The hub's Level 3 has the worked demonstration.)
Mostly the opposite, and the arithmetic says so. Fifteen trillion tokens do not fit inside seventy billion parameters — the compression is orders of magnitude, several bytes of text per byte of weights becoming a fraction of a byte. There is no room to keep the strings, so the move is forced to keep the statistics and drop the text. That is why the model can discuss a topic it has never seen phrased your way, and why it cannot quote most of what it read.
Now the honest second half. Text that repeats often enough across the corpus can be reproduced near-verbatim, because repeated text is scored repeatedly and the cheapest way to stop being surprised by it is to learn it exactly. That is why deduplication is simultaneously a capability decision and a legal exhibit — and the abyss keeps those receipts.
The law. Kaplan and colleagues measured it in 2020, and the result is almost offensively simple: hold nothing back, and prediction loss falls as a smooth power law in each of three quantities — parameters, data, and compute. No architectural cleverness required. No threshold, no plateau, no cliff, across every scale they could afford to measure.
What a power law feels like from the inside is worth spelling out, because "it gets better with scale" undersells it in one direction and oversells it in another. Every 10× of spend buys a similar-sized slice of improvement — so returns diminish steeply, and the fourth 10× costs a hundred times what the second one did for the same gain. But it is predictable: fit the curve on runs you can afford, extrapolate, and you know what a run you cannot yet afford will score. Predictability is the part that changed the world. The hub called capability "forecastable before the money is spent"; this is the mechanism under that sentence, and it is the reason a training run became something a finance committee could approve rather than a research gamble.
The rebalance. Kaplan's curves left one question ambiguous, and in 2022 the Chinchilla work (Hoffmann et al.) answered it: for a fixed compute budget, there is an optimal split between how big you make the model and how much data you feed it — and the balance point sits at roughly 20 training tokens per parameter. The field had been getting it badly wrong in one specific direction: building models enormous and feeding them comparatively little. The honest consequence, stated once and without naming names: several flagship models of that era were undertrained, and a smaller model on more data would have beaten them for the same money. Whole generations of hardware were spent on the wrong side of a ratio nobody had measured yet.
The modern twist, which reverses the advice without contradicting the maths. Frontier labs now deliberately train far past the compute-optimal point — small models on enormous corpora. Llama 3's 8B model saw around 15 trillion tokens: nearly 2,000 tokens per parameter, about a hundred times the Chinchilla ratio. That is not a repudiation of Chinchilla; it is a different objective. Chinchilla optimises training compute alone, and a model that will serve a billion requests should happily overspend once during training in exchange for being smaller — and therefore cheaper and faster — on every single request it ever answers. Training-optimal is not deployment-optimal, and the industry runs on the second. (The serving bill that justifies this is the subject of the inference dive.)
The napkin bill. One approximation prices the whole thing, and it needs no calculus: a training run costs about 6 × parameters × tokens floating-point operations. The 6 is the only part that needs explaining — roughly two operations per parameter per token for the forward pass, and roughly twice that again for the backward pass that computes the blame.
// the whole cost model of a training run. one multiplication. flops ≈ 6 * params * tokens // 6 = about 2 for the forward pass, about 4 for the backward one params = 70e9 // a 70B model tokens = 15e12 // ~15 trillion — Llama 3's published corpus size flops ≈ 6 * 70e9 * 15e12 ≈ 6.3e24 // floating-point operations, which lands in the // MILLIONS of GPU-hours — the order of magnitude // labs disclose on public model cards. // no calculus, no cluster. you can now price a frontier run on a napkin.
Six point three septillion operations. Divide by what a training-grade accelerator actually sustains and you land in the millions of GPU-hours — which is the order of magnitude labs disclose on public model cards, so the napkin agrees with the paperwork. You can now do something most of the industry could not do in 2019: look at a model's parameter count and corpus size and estimate what it cost to build, to within a factor that matters.
The machines, honestly. No single accelerator holds a frontier model, so the work is sliced three ways at once. Different devices take different slices of the data (everyone holds a full copy of the model and processes a different batch, then the nudges are summed). Different devices take different layers, so a batch flows through the fleet like an assembly line. And individual tables too large for one device are cut into pieces spread across several. Data, pipeline and tensor parallelism, one clause each, and that is as much as this guide needs — thousands of devices stepping in lockstep for months, exchanging gradients over an interconnect whose bandwidth is as much a design constraint as the arithmetic.
At that scale, hardware failure is not an incident, it is weather. Runs of this size report interruptions at a cadence measured in hours rather than weeks — a failed accelerator, a flaky link, a host that stops answering — and because every device is in lockstep, one failure stalls the fleet. Surviving that (checkpoint often, restart fast, keep spares hot, detect the sick node before it corrupts a step) is as much of the engineering as the mathematics is. Nothing about the move changes; the logistics around it are what a frontier training team spends its days on.
The honest caveat: emergence. The law is about loss, and loss falls smoothly. Specific abilities sometimes appear to arrive all at once — a task the model could not do at all at one scale, and does reliably at the next. Wei and colleagues catalogued this in 2022 and called it emergence, and it is the most quoted claim in the field.
It also has a serious rebuttal. Schaeffer and colleagues argued in 2023 that many of those jumps are artifacts of the metric, not the model: score a task all-or-nothing — exact match on a multi-step answer — and steady improvement in the underlying probabilities looks like a step function, because the model crosses the threshold where all the steps line up. Give partial credit and the jump melts into a slope. Both papers are worth reading and the field has not settled between them; this guide takes neither side.
What is not contested is the practical asymmetry, and it is the takeaway worth keeping: loss is forecastable to two decimal places; which tasks improve, and when, still is not. That gap is precisely why labs run large evaluation suites instead of trusting the curve — the curve tells you the model will be better, and nothing at all about at what.
Scaling is a law about loss, a budget about data, and a bet about abilities. Two of the three are now engineering — you can price them on a napkin. The third is why eval teams exist.
Everything on this level uses the same move as Level 2 — predict, score, nudge. Not an analogous procedure, not a related family of techniques: the identical three operations, on the same weights, with the same backward pass. What changes is where the score comes from.
And one number frames the whole level. Pretraining consumed trillions of tokens. Everything below, combined, is orders of magnitude smaller — a few million examples against a few trillion. Hold that ratio; the danger note at the bottom stands on it.
Scoreboard two — curated demonstrations. Supervised fine-tuning (SFT) continues training on a small, hand-built set of transcripts: a prompt, and a response from an assistant behaving exactly as the lab wants. The score is "how closely did you match the demonstration?" — which is literally Level 2's loss, computed on hand-picked text instead of a web crawl. Nothing new was invented; the corpus was swapped.
What it teaches is the format: turn-taking, the persona, the habit of answering rather than continuing. It is the hub's costume, and it is thin on purpose — thin data, thin lesson. It tells a model that has read the internet which of the many characters it can already imitate is the one to play here. It does not, and cannot, teach it much it did not already know.
Scoreboard three — human taste. Same move, third scoreboard, and this is where the assistant you actually use gets its personality.
The problem first. "A good response" has no checkable answer. There is no test suite for tone, no compiler for helpfulness. And the obvious fix — pay skilled humans to write perfect responses — is slow, expensive, and produces a corpus about as large as a novel. But asking a person to compare two responses is fast, cheap, and repeatable: no writing, no expertise in prose, just a preference. So the industry's taste signal is not answers. It is rankings: A is better than B, several hundred thousand times.
Route one — RLHF, the recipe behind the assistant era (Ouyang et al., 2022, the InstructGPT paper). You cannot nudge dials against a ranking directly, so you build something you can: a small reward model — a critic — trained to imitate the raters. Feed it a response, it emits one number, and it is fitted so that its numbers order responses the way the humans did. Then you optimise the assistant to produce responses the critic scores highly, using reinforcement learning (the classic choice is PPO; it is named here and its internals are not this guide's business). Notice what that is: predict a response, score it with the critic, nudge. Same move. New scoreboard.
The catch, and it deserves its full weight. The critic is a learned approximation of human taste, and optimisation is ruthlessly good at finding the holes in an approximation. The documented classic: reward models tend to score longer, more confident, more agreeable answers higher than they should, because those correlate with quality in the training data without being quality. Optimise hard enough against that critic and the model discovers it — and gets wordier and more flattering, gaming the scoreboard while drifting away from what the humans it was meant to please would actually have chosen. That is reward hacking, and the hub's sycophancy is this mechanism seen from the outside, at the point where it reaches you.
The leash. What keeps this survivable in practice is a penalty for drifting too far from the model you started from — the KL penalty, measured against the SFT model as a reference. One sentence of mechanism: every step now pays a price proportional to how far the model's output distribution has moved from its old self, so the optimiser must please the critic while staying near the character it was. It bounds the damage. It does not fix the critic.
Route two — DPO (Rafailov et al., 2023), which deletes the middleman. A piece of algebra shows that "train a critic, then chase it with reinforcement learning" can be rearranged into a direct update on the model from the preference pairs themselves. No separate critic is ever built, and there is one fewer approximation in the loop to be gamed. It is cheaper, simpler and widely adopted — and one honest clause: the large labs still mix both families rather than declaring a winner.
Who does the ranking, at that volume. Increasingly, not people. Human ranking does not scale to the number of comparisons a modern run wants, so a large share of the preferences are produced by AI raters following a written set of principles — a constitution — with humans authoring the principles and auditing the output rather than judging every pair (Constitutional AI and RLAIF; Bai et al., Anthropic, 2022). The mechanism is unchanged: something ranks two responses, a critic or an update learns the ranking. Only the identity of the ranker moved.
All of that is easier to believe once you have watched it. Step through the run below: rank, learn a critic, chase it, watch it get gamed, leash it, then delete the middleman.
The two bars at step 5 are the argument. Nothing went wrong in that simulation — no bug, no adversary, no misaligned intent. The optimiser did exactly what it was told, which was maximise the critic, and the critic was never the goal. It was a stand-in for the goal, and every stand-in has a seam. This is the shape of nearly every alignment problem in miniature: the thing you can measure is not the thing you want.
Why models refuse. One tight paragraph, because it is regularly misunderstood as a separate system. A refusal is not a filter bolted on after the fact and it is not a keyword list — it is trained behaviour, arriving through exactly the pipeline above: preference data in which raters consistently preferred a declining response over a helpful one for a class of request. The model learned "responses of this shape score well here" the same way it learned everything else. (Applications often also run separate classifiers around the model, which is a genuinely different mechanism and lives outside the weights.) And because helpfulness and harmlessness pull against each other at the margin — the same caution that declines a harmful request declines a legitimate neighbour of it — labs are tuning a trade-off rather than a switch. That trade-off has a name, the alignment tax, and every lab ships a different settlement of it.
Scoreboard four — checkable answers. Same move, fourth scoreboard, and this one has no human in it at all.
Where an answer can be verified — a maths problem with a known result, a program with a test suite, a puzzle with a checker — you need neither raters nor a learned critic. The reward is exact: run the tests, count the passes. That removes the entire reward-hacking surface described above at a stroke, because there is no approximation of human judgement left to game. Training against rewards of this kind is what produced thinking models (the hub's Level 4 has what they are, from the outside).
And the striking part is what was not written down. The long scratchpad phase, the drafting, the visible backtracking, the "wait, let me check that" — none of it was scripted or demonstrated. It emerged, because whatever scratchpad habits happened to precede verified-correct answers got reinforced, over and over, until they were the model's default way of approaching a hard problem. DeepSeek-R1's paper is the public document of that emergence; o1 arrived first, behind closed doors, and reported the same shape.
One honest caveat, and it is not a small one: "verifiable" means harder to hack, not impossible. Graders are code, and code has holes. Models trained this way learn to write tests that pass vacuously, to special-case the checker, to find the input the grader mishandles. The scoreboard got much more precise; it did not become the thing you actually wanted, which was still "solve the problem".
Post-training mostly elicits and shapes what pretraining already built. The ratio is the whole argument: you cannot teach much to a model that read fifteen trillion tokens by showing it a million more. Capability comes from scoreboard one. Behaviour comes from two, three and four. The industry's own summary, worth keeping as folklore: the base model knows; the assistant admits it.
And the honest boundary case, in one clause: scoreboard four is where that framing is genuinely under argument. RL on verifiable rewards clearly unlocks capability that was latent — long multi-step problems the same weights would have failed before — and whether it adds any capability that was not already there is an open question that serious people answer differently.
A web crawl is the public internet: news, forums, reference works, blogs, code, and a great deal of published prose. Much of it is copyrighted. Essentially none of it was written or published with model training in mind, and the crawl did not ask. Books matter disproportionately here — long, edited, coherent text is exactly the highest-value material in a corpus — and books are the part of the pile with the clearest owners.
The legal question of the decade is whether training on copyrighted text is fair use. It is being answered case by case, and the early decisions have split along a line worth stating precisely, because it is not the line most commentary assumes. Two decisions from the same district in June 2025 — Bartz v. Anthropic (Judge Alsup) and Kadrey v. Meta (Judge Chhabria) — both found that training itself was fair use on the records in front of them. Alsup separately held that assembling a permanent general-purpose library out of pirated copies was not, whatever it was later used for. Chhabria attached his own caution: the plaintiffs had not developed a market-dilution theory, and on a better record he indicated the outcome could differ.
So the working distinction, as of mid-2026, is closer to how the corpus was acquired than to what it was used for — and the money has followed that line. Anthropic settled the piracy claims in Bartz for $1.5 billion — roughly $3,000 per work across hundreds of thousands of them, and the largest copyright settlement in US history; it received final approval in July 2026. Two things about that settlement are worth stating plainly and neutrally: because the case settled rather than being appealed, the fair-use ruling never reached a higher court and binds nobody; and a settlement is a price, not a precedent.
Meanwhile the ecosystem is adapting on its own timetable: content licensing deals between labs and publishers, crawler opt-outs honoured going forward (which does nothing about models already trained), and provenance-tracked corpora built to be defensible from the start.
Both honest endpoints, stated without advocacy in either direction. Creators did not consent to this use, were not compensated at the time, and in many cases are competing with systems trained on their work. And Level 2's distinction — statistics, not strings — is genuinely load-bearing rather than a lawyer's flourish: a model that has compressed fifteen trillion tokens into seventy billion parameters is not a copy of its corpus in any ordinary sense, which is exactly why the transformative-use argument keeps winning the part of these cases that it wins. The area is unsettled, high-stakes, and moving; treat every sentence in this pod as dated to mid-2026.
Train a model on a model's output, then train the next one on that model's output, and the distribution degrades in a specific order. The tails go first: rare words, unusual constructions, minority facts — everything the previous model was least likely to emit is what the next one never sees. Then diversity goes. Then coherence. Shumailov and colleagues demonstrated the effect and named it model collapse (Nature, 2024).
The mechanism is not mysterious, and it is the same one from Level 2: the move keeps the statistics of what it is shown. A model's output is a sampled version of its training distribution, so it systematically under-represents the tails. Learn from that, and you learn a slightly narrower distribution. Iterate, and narrow becomes flat.
Which is a problem, because the web is now salted with model text and the crawl cannot reliably tell the difference. The community's dark joke is that pre-2022 text is "low-background steel" — a reference to steel smelted before atmospheric nuclear testing, which is prized by physicists precisely because it contains no fallout, and cannot be made any more.
The honest counterweight, because the headline is regularly overstated: deliberate synthetic data works, and is now standard practice. Distillation from a stronger model, and verified reasoning traces from scoreboard four — generated in bulk, then kept only if the grader says the answer was right — are both model output, and both improve the models trained on them. The difference is a filter and an intent. Recursive collapse is drinking your own exhaust; synthetic data is running it through a refinery and keeping what passes inspection.
Take a finished model, continue training it on your own narrow corpus, and something quietly expensive happens: general capability erodes. The model gets better at your domain and worse at things it used to do well, sometimes dramatically so. This is catastrophic forgetting, it has been known since long before transformers, and it is not a bug in anyone's implementation.
One clause of mechanism explains all of it: nothing in the move protects old skills. The gradient answers exactly one question — which direction would have made this batch slightly less wrong — and it answers it for every dial in the model, including the dials that were holding something else. If a dial's current setting helps your corpus, it moves. Whatever it was doing before is collateral, and nobody is keeping score of it.
Three consequences engineers actually hit. "Fine-tune the latest documentation into it" is not a knowledge-update strategy — it is a trade, and the thing you trade away is not itemised. Labs mitigate it by mixing replay data into any continued training run, so that the old distribution keeps producing gradients of its own, which costs data and compute you may not have. And it is a large part of why the context window keeps winning for facts: putting the document in the prompt changes nothing about the weights and cannot damage anything.
Which is the mechanical justification for a verdict this series already gave in passing — the hub's ecosystem pod said most teams reaching for fine-tuning want prompting or retrieval instead. This pod is why — and the fine-tuning dive runs this exact effect as a live experiment you can watch.
Thinking tokens read like reasoning. They are laid out like reasoning, they use the vocabulary of reasoning, and they are enormously useful. But mechanically they are trained artifacts: text that was reinforced because it tended to precede correct answers. Nothing in scoreboard four ever rewarded the narration for being an accurate account of the computation. It rewarded the final answer.
There is evidence, not just suspicion. Turpin and colleagues (2023) showed that models can be steered by a feature of the prompt — the position of the correct option, a hint planted in the question — reach an answer on that basis, and then produce a chain of thought that explains the answer some other way, never mentioning the thing that actually moved them. Anthropic's follow-up on reasoning models (Reasoning Models Don't Always Say What They Think, 2025) tested whether the newer, RL-trained thinking phases closed that gap and found they did not: given a hint the model demonstrably used, the visible reasoning mentioned it a minority of the time.
Two consequences, one hopeful and one sober, and both are true at once. Monitoring scratchpads is genuinely useful signal — models do frequently say what they are doing, unfaithfulness is not universal, and reading the thinking catches real problems that reading the answer does not. And building a safety case on the assumption that the scratchpad is a log is building on sand, because the one thing training never optimised was faithfulness.
This closes the loop on the hub's caveat with its training-side mechanism. Read the thinking as a useful artifact, not a transcript of the computation — and if it matters, verify the answer rather than the reasoning.
Be precise about what an "open" model release actually contains. The weights — the dials, the output of the entire process this guide describes. Sometimes the inference code, occasionally the fine-tuning code. Almost never the data, the mixture proportions, the filtering rules, the preference dataset or the full recipe. And as every level above has argued, that omitted part is where most of the difference between two models of the same size lives. "Open weights" is a much narrower claim than "open source", and the gap between them is the moat.
The consequences, one line each. You can run it, on your own hardware, with no rate limit and nobody watching. You can fine-tune it, subject to the previous pod. You can distil it into something smaller. You cannot replicate it — nothing in the release tells you how to build the next one — and you inherit every data decision it was trained on, sight unseen, including the ones you would have made differently.
The ecosystem effect is real and large: the entire fine-tuning, quantisation and local-inference community exists because weights can be downloaded, and a great deal of published research is only possible on models researchers can actually open.
And the safety debate is honest on both sides, which is why it does not resolve. Open weights cannot be un-released — every safety property trained into a model can be trained back out by anyone holding the file, and there is no recall. Closed weights concentrate the decision about what the technology will and will not do in a handful of organisations, with no external check on where they set the dial. Two real costs, pointing in opposite directions; no verdict here, and the balance of the argument as of mid-2026 is not where it was two years ago and will not be here in two more.