Four dives built the machine, filled it with numbers and taught it manners. This one answers the question your invoice has been asking the whole time. Hitting send starts two different jobs — a reader and a writer — and every number you have ever argued about, latency, throughput, the price of a token, falls out of which of them you are paying for at that moment. (If "next-token predictor" is not second nature yet, start at the hub.) Stop wherever you like — each depth is complete on its own.
You send a prompt. Some time passes. Words start appearing, one after another, at a fairly steady pace until the reply is done. From the outside that is a single event, so the natural mental model is a single job: the machine "working on" your request at whatever speed it happens to run at, and a slow reply meaning a slow model.
Underneath, one API call starts two different jobs. They share the model's weights and nothing else.
Prefill — the reading. Your entire prompt is processed at once. Every token computes its part of the machinery, every position attends to everything it is allowed to see, and all of it happens in one sweep — this is the GPU-shaped fact from dive three, now wearing its serving hat rather than its training one. However thick the brief, it is one pass of the machine. This job ends the moment the first token of the reply appears.
Decode — the writing. One forward pass per token — the hub's cadence — each pass conditioned on everything that came before it, including the tokens this job has just produced. It is serial by construction, not by implementation laziness: token N+1 cannot be computed before token N exists, because it has to read it. No amount of hardware makes the writing parallel, because the input to each step is the output of the last one.
A server thinking about your request: one job, one speed, one bill. Under that picture "which model is fastest?" is a well-formed question, a slow reply means a slow model, and every latency complaint gets aimed at the weights. All three of those are mistakes, and the last one is the expensive kind — you end up tuning the thing that was never the bottleneck.
The reader takes in your whole brief in one gulp, however thick it is. The writer types the reply at a metronome's pace, one token per tick, re-reading everything as it goes. Two jobs, two bottlenecks, two line items on the invoice — and they respond to completely different optimisations. Nearly everything in this guide is an attempt to make one of them cheaper without hurting the other.
Two numbers name the two jobs, and every engineer meets them within a week of shipping anything:
Hold those apart and a lot of confusing behaviour resolves itself. A request that takes forever to start and then streams briskly has a reading problem. A request that starts instantly and then trickles has a writing problem. They are fixed by completely different things, and Level 4 says which.
A one-sentence preview of the pricing, because it is the most-asked question about this whole subject: input tokens and output tokens carry different prices, output several times input, right across the industry. That is not a pricing strategy, it is arithmetic, and by Level 4 you will be able to derive it rather than memorise it.
And streaming, demystified in passing. Replies arrive word by word because that is decode's cadence. The interface is not dripping a finished answer out to look busy; it is showing you the metronome. A non-streaming response is the same decode with a waiting room in front of it.
One API call, two jobs: prefill reads everything at once, decode writes one token per pass. TTFT prices the reading; TPS prices the writing — and the bridge between the two jobs is a cache you have already earned.
Collect the inheritance first. At generation step 900, tokens 1 to 899 have not changed. Their vectors have not changed, so the Keys and Values those vectors produce — dive three's advertise-and-hand-over pair — have not changed either. They are bit-for-bit what they were at step 899, and at step 898, and all the way back to the moment they were first computed.
Recompute them anyway and every emitted token pays the whole n² bill again. Store them, and each decode step costs one column of genuinely new work plus one sweep over the store. That store is the KV cache, and once you have named it, prefill's real job description becomes visible: prefill is the machine filling the cache. The first output token is a side effect.
That is the bridge. The reader fills it; the writer reads it and adds one entry per tick. Everything else on this page is a consequence of how big it gets.
The mechanics, concretely. Four steps, and none of them is new machinery — this is the same forward pass you already own, bookkept:
// the hub's generation loop, with the receipts kept. K_cache, V_cache = prefill(prompt) // ONE parallel sweep over every prompt position: // this IS prefill. the first output token is a // side effect of filling the store. while not done: q, k, v = project(last_token) // one column of genuinely new work ctx = attend(q, K_cache, V_cache) // + one sweep of everything already stored K_cache.append(k) // the store only ever grows — V_cache.append(v) // nothing in it is ever recomputed last_token = sample(softmax(head(ctx) / T)) done = (last_token == STOP) // same loop as the hub's. the cache is why it is affordable.
Below, that loop has a price tag on it. Drag the two sliders to change the shape of the request, and use the three switches to take the machinery away one piece at a time — the second panel is where the argument lives.
max(arithmetic ÷ FLOP/s, bytes ÷ bandwidth); every cost is FLOPs.
Three things in there are worth doing deliberately, because between them they are the level.
Switch the KV cache off. The decode curve stops being flat and becomes a ramp, and the readout under it does the arithmetic for you: the token at position 1,000 costs about a thousand times the token at position 1. That is not a drawn shape — the widget sums the per-position attention cost one position at a time, exactly as the machine would pay it, and the quadratic falls out of the sum. This is the single most important number in serving, and it is why no production system has ever run without this cache.
Drag the prompt to 128K and watch the third panel. The cache gauge is not decoration; it is the arithmetic in the next paragraph, live. And switch GQA off while you are there, for the version of that number the industry could not afford.
Then walk the four presets. They are the four shapes of request a serving fleet actually sees, and each one puts the bill somewhere different — a long document is priced by the reader, a thinking model by the writer, and turn 40 of a chat by whether the prefix was still in the cache.
Now the memory arithmetic — this guide's census, the sibling of dive three's parameter census. Per token of context, the cache holds one K and one V for every layer, so its size is:
2 × layers × width × bytes-per-number
| 70B-class shape | Per token of context | A 128K context |
|---|---|---|
| Full attention ≈80 layers · ≈8,192 wide · 16-bit | ≈2.6 MB | ≈⅓ of a terabyte |
| Grouped-query attention same shape, 8 query heads sharing each K/V head | ≈330 KB | ≈43 GB |
Read the first row and then say it out loud: a third of a terabyte, for one conversation. Not for the model — the weights are a separate 140 GB — and not for a thousand users. For one. That number is the reason the second row exists.
Grouped-query attention (GQA) is introduced here, and it is worth saying why it was not introduced earlier: dive three deliberately kept quiet about it, because it is an optimisation that only makes sense once you know what it is optimising, and that is this table. The idea is one sentence. A layer runs many attention heads; instead of every head computing its own Keys and Values, several query heads share one K/V head. With an eight-fold grouping the cache divides by eight, and a third of a terabyte becomes forty-odd gigabytes — painful, but possible.
One honest clause about the trade: sharing keys costs a little quality, because heads that used to look at the world through their own lens now share one. The industry decided that trade essentially instantly, which tells you how badly it wanted the memory back (Ainslie et al., 2023; the single-group extreme, multi-query attention, is older — Shazeer, 2019).
Three consequences, each of which shows up in your work. Context length is a memory problem before it is a compute problem — the n² bill is real, but you run out of gigabytes before you run out of patience. The cache competes with the weights for the same device memory, which is the serving fleet's permanent real-estate crisis: every gigabyte of cache is a gigabyte not holding a model. And long chats cost more in two directions at once — linearly more per token to decode, because the sweep is longer, and quadratically more to re-read from scratch, which is why the hub's transcript-replay has a price tag attached to it.
Which brings us to the discount you have probably seen on a pricing page and never quite believed.
Prompt caching, demystified. When a provider offers cached input tokens at a large discount, what is being stored is not your text. It is the KV cache of your prompt's prefix — the computed Keys and Values, sitting in memory, exactly as prefill left them. Send a request whose beginning is identical and prefill for that span is skipped entirely: the K and V are loaded, not recomputed. That is the whole mechanism, and it explains the discount precisely — you are being charged for a memory read instead of a forward pass.
Two consequences follow immediately, and engineers hit both within a day of trying it.
The first: the match must be exact and positional. The cache is keyed to bytes at positions, not to meaning. Change one token near the beginning of your prompt and every K and V after it is wrong — not slightly wrong, wrong, because each position's vector was built from everything to its left. Everything after the edit re-prefills. A cached prefix is a prefix in the strict sense: the first N tokens, unchanged, in order.
The second falls straight out of the first, and it is the reason a piece of folk wisdom exists. If only an exact leading run can be reused, then the longest reusable run is whatever your requests genuinely have in common at the front — so putting the stable parts of a prompt first and the variable parts last is not a style preference, it is the difference between a long cache hit and none at all. Move a timestamp or a user id to the top of a system prompt and you have quietly turned the discount off for every request.
What is kept is the computed K and V for a prefix, for a short window measured in minutes, keyed to exact bytes. It is an optimisation artifact — the same numbers prefill would produce again a moment later — not a transcript store, and it exists because recomputing it costs more than holding it. (What a provider retains, for how long, and for what purpose is a separate and contractual question; it is not answered by this mechanism in either direction.)
The trap's useful half is the opposite of the fear. Most teams underuse caching, and almost always for the same reason: they put the variable content first — the retrieved documents, the user's name, today's date — and break the prefix before it starts. Fix the ordering and the discount appears without another line of code.
Every pass ends the same way: a distribution over the whole vocabulary. The hub's sandbox let you feel that — bars, mass, a slider. What it did not show is that serving systems do not sample from that distribution as it comes out. They interpose a pipeline of reshaping steps between the model and the dice, and the order of operations is fixed.
// what sits between the forward pass and the dice. logits = model(tokens) // one raw score per token in the vocabulary probs = softmax(logits / T) // TEMPERATURE — reshapes everything, deletes nothing probs = top_k(probs, k) // keep the k highest. deletes: everyone else. BLUNT probs = top_p(probs, p) // keep the smallest set summing to p. ADAPTIVE probs = probs / sum(probs) // renormalise — the survivors are the new 100% tok = sample(probs) // the only randomness in the whole system // order matters — every serving stack runs some version of this pipe, // and the dials are the API parameters you already use.
The zoo, one dial at a time. Each one does exactly one thing to the distribution, and it is worth learning them as operations rather than as vibes.
Why the zoo exists at all — and this is the level's intellectual payoff, because "add randomness so it is less boring" is not the reason.
Pure greedy decoding — always take the most probable token — produces degenerate text. Not merely dull: repetitive, looping, weirdly flat, prone to falling into a phrase and saying it again and again. That is a measured property, not a matter of taste. And the obvious alternative, sampling honestly from the full distribution, trips over the other end: the tail of a vocabulary is a hundred thousand tokens no sane continuation uses, and roll enough dice and one of them wins.
Holtzman and colleagues named the dilemma in 2019, in a paper whose title says it: The Curious Case of Neural Text Degeneration. The head of the distribution is too narrow to be interesting; the tail is too unreliable to touch. Every dial in the zoo above is a different way of drawing the line between them — and nucleus sampling won because it draws that line where the distribution itself says it should go.
Below, the pipe runs. Two tabs, because the contrast is the lesson: the same dial, at the same setting, on the two shapes of distribution a sampler actually meets.
The pair of numbers in the census line is the whole argument. Leave top-p at 0.90, flip between the tabs, and watch the count of survivors change without you touching anything — a handful of candidates on the near-forced token, roughly ten on the open one. No fixed k can do that, because k does not know which distribution it is standing in front of. Then turn top-k on and watch it guillotine the flat tab and the spiked one to the identical number, which is precisely the failure.
Two other things are worth a deliberate look. Drag temperature up and watch the tail row swell before any truncator gets a chance — that is the mechanism behind text falling apart above T ≈ 1.5, and it is also why temperature and top-p are not redundant with each other. And press Sample ×100: the histogram against the final bars is the reminder that all of this reshaping is in service of one roll of the dice, and the pipe's only job is to decide what the dice are allowed to land on.
Greedy decoding is not "most correct." It is most-probable-at-each-step, which is a different thing — and the hub's no-backtracking fact is why the difference bites: a sequence of locally safest tokens can paint the sentence into a corner that no later token can get it out of. Looping is the visible symptom of exactly that.
For verifiable tasks, low temperature buys you reproducibility, which is genuinely valuable and is not the same as truth. Determinism is a property of your test suite, not of the answer. (And it is not even fully deterministic across deployments: batching changes the order floating-point numbers are added in, and floating-point addition is not associative. Identical prompt, identical weights, occasionally a different token. Do not build anything that depends on bit-exact replies.)
The mechanical postscript on streaming. Tokens leave the sampler one at a time, so streaming costs a provider nothing to offer — the tokens exist individually whether or not anyone forwards them. The only real question is whether the interface shows you the metronome or hides it, and a buffered "non-streaming" response is the identical decode with a waiting room. Which is also why "streaming made it faster" is an illusion worth naming: the last token arrives at the same moment either way. What changed is when the first one did.
Everything so far has followed a single request through the machine. That is the wrong unit. No provider runs one request at a time, and the reason is not efficiency in the vague sense — it is that a lone request wastes almost all of the hardware, for a reason the abyss states as physics and this level states as a bill.
Batching — why providers stack strangers on your machine. Here is decode's dirty secret. Generating one token requires streaming the model's entire weights through the chip once, and the amount of arithmetic done per byte fetched is tiny. The chip spends its time waiting for memory, and its arithmetic units — the expensive part — sit mostly idle.
But the weights streamed once can serve many requests' next token in the same sweep. Fifty conversations decode together for very nearly the price of one, because the expensive thing was the streaming, and they shared it. That is the entire economics of LLM serving in one sentence, and it is why the industry's answer to "make it cheaper" has been "find more people to put on the same machine" rather than "make the model faster."
The modern refinement is continuous batching: requests join and leave the batch mid-flight rather than waiting for the slowest member of a fixed group to finish. It is the serving-stack standard, and it has a consequence you have almost certainly observed without diagnosing — your tokens-per-second varies with the provider's load while your bill does not. You are being charged per token, and per token you are getting a share of a machine whose occupancy you cannot see.
Quantisation — fewer bytes, same dials. Store the weights (and, increasingly, the KV cache) in 8 or 4 bits instead of 16. The capacity win is obvious: half the bytes, half the memory, twice the model or twice the cache in the same box.
The speed win is the subtle one, and it is the more valuable half. Decode is paced by byte-streaming, not by arithmetic — so halving the bytes very nearly halves the tick. Quality cost: small at 8-bit, real and contested at 4, and the abyss goes lower. One framing sentence worth carrying out of this level, because it explains this dial and the last one and the next one: the bytes are the bill.
Speculative decoding — the elegant trick. A small, fast draft model types ahead, guessing the next k tokens cheaply. The big model then verifies all k in one parallel pass — which is prefill's trick, aimed squarely at decode's problem, since checking k tokens you already have is a reading job. Where the draft guessed right, you got k tokens for one sweep of the big model's weights. Where it guessed wrong, the big model's own choice stands from the first mismatch onward and the rest of the draft is discarded.
The load-bearing precision here is the same one dive three insisted on for FlashAttention, and it deserves the same emphasis: done correctly, the output distribution is provably identical to the big model decoding alone. Not similar, not close enough, not a quality trade you have to think about — identical, by construction, with a proof (Leviathan et al., 2023; Chen et al., 2023). It is a pure speed trick. It pays best on predictable text — boilerplate, code, structured formats — where a small model guesses right for long runs, and it pays least on the prose where every token is a real decision.
The invoice, assembled. Four questions, and every one of them is now a derivation rather than a fact to memorise.
Why do output tokens cost several times input tokens? Because prefill amortises one parallel sweep across your entire prompt — a thousand input tokens share one pass of the machine — while every single output token pays for its own full forward pass and its own sweep of the weights. The reader gets a bulk rate. The writer pays retail, one token at a time, forever.
Why is cached input the cheapest line on the bill? Because it involves no compute at all. The K and V already exist; the machine loads them and moves on. You are paying for storage and a memory read, which is why the discount is large rather than polite.
Why does long context hurt twice? Because it hits both jobs. The reader pays n² at prefill — the whole prompt attending to the whole prompt. The writer then pays for the resulting cache in gigabytes, on every step, for the rest of the conversation. Level 2's third panel is that second cost, and it is the one people forget when they reach for a bigger window.
Why do thinking models bill like enormous outputs? Because that is exactly what they are. Thinking tokens are decode tokens — same loop, same cadence, same price — and "think harder" is literally "buy more of the most expensive line item." The hub's test-time-compute knob and dive four's fourth scoreboard both described this from the inside; here is its price. There is nothing mysterious about a reasoning model's bill: it wrote a great deal, most of which you never saw.
It is a property of the deployment. Batch depth, quantisation, cache policy, hardware generation, how many other people are on the machine, and which serving stack is running will move the same weights across an order of magnitude of throughput — without anyone touching a single parameter.
So if you mean to compare deployments, benchmark deployments, under your own traffic shape, and say so in the write-up. Benchmarking "the model" by timing an API from your laptop mostly measures who else was on the machine that afternoon. The same caution applies to your own latency dashboards: a TPS regression is far more often a batching or load story than a model story.
This is the napkin that explains the entire level above, and it is two numbers and a division.
A 70B model in 16-bit is about 140 GB of weights. Decode streams all of them, once per token, at batch size one — there is no way around it, because every parameter participates in every forward pass. Top-tier accelerator memory moves a few terabytes per second. Divide one by the other and you get a ceiling of a few tens of tokens per second, and that ceiling holds no matter how fast the arithmetic is.
Now look at the other side of the ratio. The arithmetic in that same forward pass is roughly two operations per parameter — call it a few hundred billion operations — which an accelerator-class chip can do in well under a millisecond. The memory took tens of milliseconds. The chip's maths units could go a hundred times faster than its memory can feed them, and during decode they simply wait. That is not a tuning failure; it is the shape of the problem.
Every trick in Level 4 is this one ratio being fought from a different angle. Batching amortises the stream across many users, so the same bytes produce fifty tokens instead of one. Quantisation shrinks the stream, so there are fewer bytes to wait for. Speculative decoding buys more tokens per stream, by having a cheap model guess what the expensive one was going to say. Three answers, one question: how do we get more out of one pass of the weights?
And the contrast that closes the loop back to Level 1: prefill is arithmetic-bound. It reads the weights once too, but it does a thousand tokens' worth of work with them, so the ratio flips and the maths units are the constraint. The two jobs are not merely different in software — they are limited by different physical properties of the same chip, which is why they price differently all the way down to the silicon. (All figures approximate and accelerator-class; the point is the ratio, and the ratio has held across several hardware generations.)
Level 2's cache is simple in principle and vicious in production, for a reason every systems engineer will recognise before the paragraph ends.
Conversations grow unpredictably. You cannot know how long a reply will be until it ends, so a serving system either reserves the maximum — wasting enormous amounts of memory on requests that stop after forty tokens — or reallocates as it goes, and fragments. Naive implementations wasted a large fraction of the memory they had, which in a business where memory is the constraint is the whole margin.
The fix, published as PagedAttention in the vLLM paper (Kwon et al., 2023), is an answer out of an operating-systems textbook: cut the cache into fixed-size blocks, map them through an indirection table, and let a sequence's blocks be scattered anywhere in physical memory. Fragmentation disappears because there is nothing left to fragment — every allocation is one block. Every systems engineer reading this has seen this movie: it is virtual memory, for attention, and it became the default idea in serving stacks almost immediately.
The second half of the trick is the one that closes Level 2's loop. Once blocks are indirected, two requests whose prompts begin identically can point at the same blocks instead of each holding a copy. That is prefix sharing, and it is prompt caching's engine room — the same mechanism that gives you a discount across your own requests gives the provider a memory saving across everybody's. The pricing page and the paging table are the same idea, seen from opposite ends.
Quantisation's frontier, stated as honestly as the evidence allows, which is less confidently than the internet does.
8-bit is a solved problem and close to a free lunch — the quality difference is small enough that it is routinely used without comment, and the capacity and bandwidth wins are immediate. 4-bit weights are the working standard for local inference, with real but usually acceptable loss; this is the setting most people run at home, and it is the reason they can. Below that — 3-bit, 2-bit, mixed schemes that keep some layers wide and squeeze others — quality falls off a cliff whose exact edge is contested, model-dependent, and moves with every new method. Treat any confident number in that region as a claim about one model on one benchmark.
Two cautions that matter more than the headline. The first: quantisation loss is uneven. It does not degrade everything by a percent; it shows up in the tails — rare knowledge, edge cases, long chains of reasoning where a small early error compounds — well before it shows up in an aggregate benchmark score. A quantised model that scores the same can behave noticeably worse on precisely the hard cases you deployed it for.
The second: the KV cache is quantisation's other frontier, and given Level 2's table it may be the more valuable one. The same bytes-are-the-bill logic applies — a cache in 8-bit is half the gigabytes and half the bandwidth per decode step — and the same unevenness applies to its risks.
This is an open trade, not a verdict. Where you set the dial depends on what you are running, where you are running it, and which failures you can tolerate — which is the same shape of answer dive two gave about tokenizers, and for the same reason: the industry is still paying rent on a decision nobody has been able to settle.
This series has taught the same knob twice from two different directions: the hub gave the mechanism — generated tokens are the only working memory a model has beyond one fixed-size pass — and dive four gave the training recipe that turned that trick into a habit. This pod gives the price, which is the part that decides whether you use it.
The price is simple to the point of bluntness: thinking tokens are decode tokens. They stream through the same loop, at the same tokens-per-second, billed on the same line, and they are the expensive line. A model that thinks for two thousand tokens before answering has bought two thousand tokens of the most expensive thing on the invoice. Sometimes that is the best money you will spend all week; often it is being spent on a request that a single-hop lookup would have answered.
The frontier is pushing in two directions, and they cost differently. Serial: think for longer — one chain, more tokens, the o1 and R1 lineage. Latency grows with the spend, and there is a point past which the chain stops helping. Parallel: sample several independent attempts and pick between them by voting or by verifying — best-of-N. Latency stays roughly flat because the attempts run at once, the bill multiplies instead, and it pays exactly where answers are checkable, which is dive four's graders showing up again, this time at inference.
Which creates the routing question every engineer building on these systems now owns: which requests deserve the spend? Getting it wrong is expensive in both directions — thinking on trivia burns money, not thinking on a genuinely multi-step problem burns accuracy — and the honest observation, as of mid-2026, is that the industry is still learning to route. Automatic routers exist, they are getting better, and none of them yet relieves you of having an opinion about your own traffic.
The series' parting gift, and it falls straight out of pod 1's arithmetic — which turns out to be unexpectedly kind to the smallest possible deployment.
A single user needs one stream of the weights per token, and nobody else is queuing. So the only question is how many bytes your machine can move per second and how many bytes the model is. A 7B model at 4-bit is about 4 GB. That fits in a consumer machine's memory with room to spare, and consumer memory bandwidth divided by four gigabytes lands at a perfectly usable tokens-per-second — the same division as pod 1, on numbers three orders of magnitude smaller, coming out fine. The thing that makes batch-1 serving uneconomic for a provider is exactly what makes it viable for you.
The ecosystem grew around that fact. llama.cpp and its many descendants run quantised models on ordinary hardware, and they run the open-weight models from dive four's last pod. What you give up is real: frontier quality, and the big-batch economics that make an API cheaper per token than your own electricity. What you get is also real: privacy, no rate limit, zero marginal cost, and — the reason it belongs at the end of this series — the entire serving stack on your desk, inspectable.
Which is the best last exercise this series can assign. Everything these five dives described is running on that laptop and can be printed: the tokenizer splitting your sentence into chunks, the embedding table handing over a row per chunk, attention gathering across positions, the KV cache growing by one entry per tick, the sampler drawing from a distribution you can dump to the terminal. It is all there, it is all small enough to read, and none of it is magic. It never was.