The hub called it model(tokens) and moved on. The last dive left every word wearing the same vector in every sentence it ever appears in. This descent opens the box between the two: the machine that turns context-free vectors into context-full ones, one attention step and one MLP step at a time, dozens of times over. (If "next-token predictor" is not second nature yet, start at the hub.) Stop wherever you like — each depth is complete on its own.
First, the ground this dive stands on — two facts, each with a whole dive behind it. An LLM does one thing: read a sequence of tokens and score every candidate for the next one. That prediction, run in a loop, is the entire product — the hub wrote the step as model(tokens) and treated the model itself as a black box. And by the time that black box sees them, tokens are not text: each has been swapped for a vector — a long list of numbers looked up from a fixed embedding table (the last dive). Fixed is the trouble. The table hands over the same row for bank in "river bank" and "bank deposit", because every row was written during training, long before your sentence existed. This descent opens the black box — and the machine inside has exactly one job: make each token's vector absorb its context.
Make it concrete. In The cat sat on the mat because it was tired, the vector for it has to come to contain cat-ness — to drift measurably closer to cat than to mat, in the meaning-as-geometry sense the embedding dive's own deep levels develop — or the next token is a guess. No lookup table can do that, because the table was written before the sentence. The machine has to build it, live, on every pass. And notice the shape of the job: the word that settles what it means sits somewhere else — three words back here; in real text, three paragraphs or three chapters back. Contextualising a token means moving information between positions, possibly very distant ones. Hold that requirement; it is about to break something.
So how would you build that machine? The pre-2017 answer was the obvious one: read the way a person reads. A recurrent network (RNN) walks the text left to right carrying one running summary vector — at each word, fold the new word into the summary, hand the result to the next step, like a relay runner passing a baton. The baton is a fixed size: one vector, whether the document is ten tokens long or ten thousand. Everything the network knows about the text so far has to fit inside it.
Challenge one: the baton never gets bigger. A fixed-size summary of a growing document must keep forgetting — that is arithmetic, not an engineering oversight someone could have patched. For word 12 to still be present in the summary at word 900, every one of the 888 steps in between has to choose to keep it over something newer. Long-range information dies in the retelling, and the pronoun three chapters from its noun — the exact case the job demands — is the first casualty.
Challenge two: reading in order is a queue. Step 900 cannot be computed until step 899 has finished — not slow, impossible, by construction. And the hardware that made deep learning affordable, the GPU, is the opposite shape: thousands of small calculators that pay off only when handed thousands of independent jobs at once. Give a GPU a queue and nearly all of it stands idle. So recurrent networks were not merely forgetful — they could barely use the one machine capable of training them at scale, and that put a ceiling on how big they could ever get.
Read in order; keep one summary; pass it on. Every fact about the beginning of a document has to survive being copied, hundreds of times, through a memory that never gets any bigger. Long documents become a game of telephone — and because each step waits for the one before it, the whole thing trains at the speed of a queue.
Everyone is in the room at once. At each layer, every token looks directly at every other token that came before it — no summary in between, no relay — decides which of them matter to it, gathers what it needs, and updates its own notes. Then the next layer does it again, with the notes improved. Nothing has to survive a retelling, because nothing is retold — and because no token waits for any other, the whole room computes at once. Both challenges, gone.
And the whole architecture is two moves, alternated. Each layer of the machine does exactly this much:
Communicate, then compute. Do it dozens of times, on a running vector per token. That is a transformer; everything below this level is the mechanism, the arithmetic and the bill.
That is not how the story actually went, of course — nobody designed the conference room from scratch. Attention was invented as a patch: Bahdanau and colleagues (2014) let a recurrent translation network peek back at the source sentence while writing each output word, instead of trusting the baton to have carried everything. The patch worked embarrassingly well. Three years later, "Attention Is All You Need" (Vaswani et al., 2017) made the move its title announces, literally: delete the recurrence, keep only the peeking. Translation got better — and training became massively parallel, because with the relay gone, nothing queues. Level 4 explains why that second clause, not the first, is the one that conquered the world.
A transformer alternates communicate (attention: information moves between tokens) and compute (the MLP: each token thinks alone) — dozens of times. The old answer, recurrence, failed the job twice — a memory that had to forget, and a queue that could not use the hardware. The conference room answers both.
Take one token, in one layer. Its vector at this point is the running total of everything so far — at layer one, that is the embedding row the last dive handed over, with position mixed in. The layer multiplies that vector by three learned tables and gets three smaller vectors out. (A "learned table" is exactly what it sounds like: a grid of numbers fitted during training and frozen afterwards — the same status as the embedding table, no more mysterious.)
The three have names, and the names are the whole idea:
The match is a dot product — the same "similar directions mean related things" fact the embedding map taught, except that there it compared two fixed rows of a table and here it compares two vectors manufactured this very pass, from this very sentence. Every query is scored against every key. The scores are then divided by a constant to keep them in a useful range for what comes next (that is the "scaled" in scaled dot-product attention — one line of arithmetic, no anxiety required).
Then softmax — the hub's function, the one that turns logits into next-token probabilities — does its second job in this machine. It turns a row of match scores into a distribution over who gets heard: all positive, summing to one, with the strongest matches taking most of the mass.
The last step is the one people skip, and it is where the information physically moves. Each token takes the weighted average of the Values of the tokens it attended to — 45% of that one, 40% of that one, a trickle of everything else — and adds what it gathered to its own vector. That is attention, complete. A token asked a question, everyone answered, and the answer is now part of the asker.
One rule constrains all of it: the causal mask. A next-token predictor must not read the future, so each token may only attend to positions at or before its own; later positions are struck out of the row before softmax runs, and therefore receive exactly zero weight. This single rule is why the machine can be trained on "predict every next token of every document, all at once" (Level 4 collects that payoff) — and it has a consequence worth holding onto: a token's vector can only ever be built from evidence to its left.
// one attention head, for one token, in one layer q = x · Wq // three learned tables turn one vector into three: k_all = X · Wk // what I want · what I advertise · what I'd hand over v_all = X · Wv // (x = my vector; X = everyone's) scores = dot(q, k_all) / scale // guide 100's dot product, hired as a matchmaker scores[after_me] = -inf // the causal mask — no reading the future weights = softmax(scores) // the hub's function, second job: who gets heard gathered = weighted_sum(weights, v_all) x = x + gathered // written back onto the bus — ADDED, never replaced // that is a head. a layer runs a dozen of them side by side, and the // weights above are recomputed from scratch on every single pass.
Ten lines are one thing. Below, it runs. The sentence is the transformer literature's favourite demonstration, with one word you can change. Click any token to make it the query and watch where its attention goes — and watch the mask strike out everything to its right.
Three things are worth doing deliberately in there, because between them they are the level.
Click it. Its attention is genuinely split between cat and street — roughly 45 against 40 — and it stays split no matter which word you put at the end. That is not the head being weak. At that position the sentence has not yet disambiguated anything, and the mask means it cannot peek ahead to find out. Now click the last word, and flip the toggle: tired binds it to cat, wide binds it to street, same position, same head, one word changed. The ambiguity resolves only at the token where the evidence exists — which is the causal mask made visible, and a quiet re-teaching of the hub's "no backtracking" from the inside.
Switch to head B. It is boring on purpose: every token dumps most of its attention on the token immediately before it, regardless of what the sentence says. Heads specialise, and the dull ones are not the useless ones — the induction circuit in this guide's abyss is built out of exactly this primitive.
Watch the mechanism strip. Raw scores in, masked positions removed, softmax over what survives, and a blend of Values in exactly those proportions. There is no other machinery. Attention is that strip, run for every token, in parallel.
Careful — this one costs interviews. The projections are learned: the Q, K and V tables are fitted during training and frozen thereafter. The attention pattern — who attends to whom, and how much — is computed fresh on every forward pass, from your actual input. Attention weights are activations, not parameters: the same head produces a completely different pattern on a different sentence, which is the entire point. A static table could never adapt to the data; a lookup computed on the fly can.
Multi-head attention — a panel, not a monolith. A layer does not run one big attention. It runs many small ones in parallel: GPT-2 small runs 12 heads, each working in a 64-wide slice of the 768-wide vector, each with its own Q, K and V tables, each free to learn a different kind of looking. One head tracks the previous token. One tracks matching brackets or quotation marks. One does the pronoun-binding you just watched. Most defy naming entirely. Their gathered results are concatenated back into one vector and mixed through one more learned table. The widget's head switcher was a preview of this; here it has its proper name.
Then the MLP — the half everyone skips. After attention has gathered, each token's vector goes through a small two-layer feed-forward network privately: expand to roughly four times the width, apply a simple nonlinearity, project back down. No neighbours, no communication, nothing crossing between positions. Two facts make this the underrated half of the machine:
Which gives the slogan worth carrying out of this level: attention routes; the MLP knows.
The residual stream — the shared bus. This is the centrepiece idea, and it is one word: add. Neither sub-block replaces a token's vector. Each adds its output to it. Take the vector, compute the attention update, add it back. Compute the MLP update, add it back. So a token's vector is a running total — a bus that every layer reads from and writes increments onto, and that carries the token from the embedding table all the way to the exit.
Two payoffs fall straight out of that. Gradients can flow backwards through eighty layers of additions without dying on the way, which is what makes deep stacks trainable at all (that story belongs to the training dive). And later layers can build on, sharpen, or overrule what earlier layers contributed — because everything anyone wrote is still on the bus, available to be read. (One sentence on LayerNorm, which sits before each sub-block: it is a thermostat, keeping the numbers in a healthy range so the next step behaves. Do not dwell on it.)
Below, one token's vector goes through the whole thing — read the bus, attend, add, digest, add, next layer — and then out the exit.
The stack is that block, repeated. GPT-2 small ran 12 layers on a 768-wide vector; models in the 70B class run around 80 layers on vectors around 8,000 wide. Depth is composition: a head in layer 9 can attend based on something layer 3's MLP wrote onto the bus, which no single layer could have computed. The abyss has a two-layer team caught doing exactly this.
One ingredient is still missing, and it is a big one. Attention as described is order-blind. Look at the arithmetic again: every query is scored against every key, and nothing in that operation knows which token came first. Shuffle the sentence and every dot product is identical — you would have a bag of tokens, not a sentence. So position has to be injected as data. Early transformers literally added a position-pattern to each embedding before layer one. The modern standard (rotary embeddings, or RoPE, Su et al., 2021) instead rotates each query and key by an angle proportional to its position, which makes "how far apart are we?" directly visible to the dot product. No trigonometry required here; the takeaway is only this: word order enters the machine as an input, not as an assumption.
Now the question everybody actually asks: where do 70 billion parameters sit? The answer is two lines of multiplication. Per layer, with a bus d numbers wide: the attention tables (Q, K, V and the output mix) cost about 4·d², and the MLP — up to 4d and back down — costs about 8·d². Everything else in a layer is rounding error. So per layer it is about 12·d², and the MLP really is two thirds of it.
// where the parameters actually are. two lines of multiplication. d = 768 // the width of the residual stream — the bus layers = 12 vocab = 50257 per_layer = 4*d*d + 8*d*d // attention ≈ 4d² · MLP ≈ 8d² → the MLP is TWO THIRDS stack = layers * per_layer // 12 × ≈7.1M ≈ 85M embed = vocab * d // 50,257 × 768 ≈ 39M total = stack + embed // ≈ 124M ✓ — that is GPT-2 small, to the nearest million // the same two lines, three orders of magnitude up: // d ≈ 8192, layers ≈ 80 → ≈0.8B per layer, ≈64B in the stack // the brain is not one big thing. it is one modest block, photocopied.
| Shape | Per layer 12·d² | The stack | Embedding table | Total |
|---|---|---|---|---|
| GPT-2 small d = 768 · 12 layers | ≈7.1M | ≈85M | ≈39M | ≈124M ✓ |
| 70B class d ≈ 8,192 · ≈80 layers | ≈0.8B | ≈64B | ≈1B | ≈65–70B |
Read the two rows against each other. At GPT-2's size the embedding table is nearly a third of the model; three orders of magnitude up, the same table is a rounding error and the stack is the entire budget. (The published shapes vary in the details — modern variants tweak the MLP ratio and the attention layout, so treat every ≈ as load-bearing — but the shape of the answer does not move.) The punchline is worth saying plainly: the "brain" is not one big thing. It is the same modest block, photocopied dozens of times, and nearly all of its weight is multiplication tables.
One block = attend (communicate) + MLP (compute), both added onto the residual stream. A model is that block photocopied dozens of times — and two thirds of its weight is the half nobody talks about.
Here is one forward pass, start to finish, with nothing left in the box.
Your tokens arrive as rows of the embedding table — one fixed vector each, context-free, exactly as the last dive left them. Position is mixed in. Then layer one: every token derives its query, key and value; every query is scored against every key it is allowed to see; softmax turns those scores into weights; each token gathers a blend of Values and adds it onto its bus. Each token then digests privately through the MLP and adds that on too. Layer two does it again, on vectors that now carry context. And again, and again, through the stack — bank drifting riverward or vaultward, it becoming cat-flavoured at exactly the position where the evidence allowed it.
At the end, the vector at the last position meets the unembedding table — the embedding table's mirror at the exit — and becomes a score for every token in the vocabulary. Logits. Softmax. Sample. You are standing at step 2 of the hub's four-step loop, and you have just been inside model(tokens).
Now the price tag. Three consequences, each derived from the assembly above rather than asserted — and each one shows up on an invoice or in an incident review.
1 · Why this architecture ate the world: it is GPU-shaped. Look at what the mask does during training. Every position's computation is independent of every other position's — the mask hides the future from each token, but all tokens compute at once. So a single pass over a document trains the model against every next-token target in that document simultaneously, thousands of predictions from one sweep of the hardware. The relay could never do this: position 900 had to wait for position 899, by construction. A transformer turns a sequence into a matrix multiplication, and a matrix multiplication is precisely what a GPU is.
They won because they train in parallel on the hardware that happened to exist. That is the honest mechanism, and the order matters: parallelism bought scale, scale bought quality, and the scaling laws turned that into a budget line an executive could sign. The architecture did not out-think its predecessors — it fit the machine. Capability followed, and it followed because a hundred times more compute became spendable, not because anyone found a cleverer idea about language.
2 · The n² bill. Every token attends to every earlier token, so the number of score-and-gather operations grows with the square of the sequence. Ten times the context is roughly a hundred times the attention work when the prompt is first processed. This is the structural reason long context is expensive, and why "just make the window bigger" is never free — the abyss tours the escape attempts, and the full cost model belongs to the inference dive.
3 · Fixed compute per token — now mechanically grounded. The hub said multi-step arithmetic fails because each token gets one fixed-size forward pass. You can now see the fixed size: the same stack, the same width, the same twelve or eighty layers, whether the next token ends a limerick or completes a proof. There is no branch that runs the block a few extra times because the question is hard. The only variable compute a transformer has is more tokens — which is exactly why scratchpads and thinking models work at all (the hub's Level 4 has that story). And since one forward pass produces one token, generating a reply means re-reading everything before it, once per token — except that the Keys and Values of the positions that have not changed do not change either, so they can be cached and reused. Four words that are the entire subject of the inference dive; they are not expanded here.
The pattern is the simplest form of "continue what you have already seen": somewhere earlier in the context, token A was followed by token B; A has just appeared again; predict B. Copy-and-continue, the primitive behind reciting a name you were told two paragraphs ago in exactly the format you were told it.
Olsson and colleagues (Anthropic, 2022) traced this to a two-head team spread across layers, and the mechanism is Level 3's composition idea caught in the act. A previous-token head in an early layer — precisely the dull head B in the widget above — stamps each token with "here is what came before me", writing that onto the residual stream. An induction head in a later layer reads those stamps: its query is effectively "where did I appear last time?", and because the earlier head already wrote the answer onto the bus, it can attend to that place and copy off whatever followed. Neither head can do it alone. Layer 9 building on what layer 3 wrote is not a metaphor here; it is the wiring diagram.
Two things make it matter beyond the neatness. These heads emerge abruptly during training, with a visible bump in the loss curve as they form — one of the few places where a capability and a mechanism have been watched appearing together. And they are a leading candidate substrate for in-context learning: the hypothesis (presented as a hypothesis, which is how the paper presents it) is that copy-and-adapt is the primitive underneath "few-shot examples just work".
The hub's interpretability pod promised real algorithms rather than vibes inside the weights. This is the exhibit — and the discipline that found it has a dive of its own.
A space that is d numbers wide has exactly d perpendicular directions in it. A language model needs to represent vastly more distinct features than that — every topic, register, syntactic role, entity type and idiom it has ever met. It cannot give each one its own axis, so it does not: it packs them in at angles, accepting a little interference between features that rarely co-occur. Elhage and colleagues (Anthropic, 2022, "Toy Models of Superposition") built small systems where this behaviour can be watched happening on purpose.
This is the mechanism behind an observation practitioners have made for years: single neurons are famously polysemantic. One unit fires for Shakespeare and HTTP headers and a particular kind of legal boilerplate, and no amount of staring at it yields a story, because it was never one feature to begin with — it is a corner where several are stacked.
It also explains why the sparse-autoencoder features from the hub's interpretability pod had to be extracted rather than read off neuron by neuron. The practitioner's takeaway is a correction of intuition: the model's unit of meaning is a direction, not a neuron — and there are more meanings in there than there are axes to put them on. (The interpretability dive trains one of Elhage's toy models live in your browser, so you can watch the packing happen.)
The lens above looks interpretable. Resist the promotion. A big weight on cat tells you information flowed from that position — it does not tell you the model decided anything on that basis, and it is not a reason you can put in front of an auditor.
Three reasons the inference does not hold. The Value vectors transform what is carried, so a large weight on a position whose Value contributes little means little. Heads cancel, duplicate and overrule each other, and what matters is the sum on the bus, not one summand. And whatever gets written can be rewritten by any of the layers above it. The literature has an unusually pointed exchange about this: Jain and Wallace's "Attention is not Explanation" (2019) and Wiegreffe and Pinter's "Attention is not not Explanation" (same year) are both worth reading, and this guide takes neither side.
The engineering consequence is concrete: never ship "the model attended to X" as a compliance-grade justification of an output. Attention maps are a fine hypothesis generator and a poor explanation. Faithful explanation of a model's output is an open research problem — the attribution-graph work in the hub's interpretability pod is the serious end of it.
Quadratic attention is the tax the whole industry is trying to avoid paying, and the escape attempts sort into three honest categories.
Approximations — sparse patterns, sliding windows, linear-attention kernels. Do not let every token see every token: give it a window, a stride, a handful of global anchors, or replace the softmax with something that factorises. These trade quality for cost, they work better on some workloads than others, and they keep being retried at every new scale because the prize is enormous.
FlashAttention (Dao et al., 2022) is a different kind of win, and the distinction is load-bearing: it computes exactly the same mathematics, reordered around how GPU memory actually moves, so that the big intermediate score matrix never has to be written out to slow memory at all. Same numbers, far less memory traffic, no approximation anywhere. That is precisely why it did not have to win an argument — it simply became the default.
The RNN's revenge — state-space models. Mamba (Gu & Dao, 2023) brings back Level 1's fixed-size running summary, but with a formulation that is both trainable in parallel and content-aware about what to keep, at linear cost in sequence length. The relay, rebuilt by people who had learned the lesson.
Why full attention still holds the frontier: the exact lookup is what buys the in-context abilities the entire product category rests on, and giving up precise recall of arbitrary earlier tokens costs exactly the behaviour customers notice. As of mid-2026 the working compromise is hybrid stacks — a few full-attention layers among cheaper ones — which is a trade, not a verdict. Treat this the way the last dive treated the tokenizer: an open question the industry is paying rent on.
Level 4 made the observation in passing; here it is in full. At generation step 900, tokens 1 to 899 have not changed. Their vectors have not changed, so their Keys and Values have not changed either — they are bit-for-bit what they were on the previous step, and on the one before that.
Recompute them anyway and you pay the n² bill on every single token you emit. Cache them, and each new token costs one column of genuinely new work plus one sweep over the stored Keys and Values. That is the difference between a chatbot that is usable and one that is a science project.
That cache has a size, a price per token of context, and a starring role in why providers charge differently for input and output — all of it the territory of the inference dive. One closing sentence you can take now: "inference optimisation" is mostly "attention-bill management".