How interpretability
really works

Start with the fact that makes this field exist. A transformer's architecture was designed by people — the attention, the layers, all of dive three. But the architecture is an empty grid of billions of dials, and everything the model actually knows how to do lives in the numbers gradient descent put there. No engineer chose those numbers. No one can point to the addition in "the part that does addition", because no one built a part that does addition. The capabilities grew, under optimisation pressure, the way a riverbed grows under water — and what is left afterwards is a program nobody wrote, in a language nobody speaks.

So interpretability is reverse engineering: treat the weights as a compiled binary and decompile. The strand this guide follows — mechanistic interpretability — wants the actual mechanism: which components compute what, composed how. (An older strand asks the narrower question "which parts of the input mattered?" — saliency maps, attribution scores. Useful, but it explains single answers, not the machine.)

Here is the strange part: the difficulty is not secrecy. Compare the neuroscientist's position. A brain is measured through a skull, a few electrodes at a time, on a subject that never does the same thing twice. A model is the opposite of that in every way — you can read every activation at every layer for every input, freeze the whole system mid-computation, replay an identical input a million times, and edit any connection to see what breaks. Perfect observability, perfect repeatability, god-mode intervention. And with all of that, the field is still hard. Which tells you where the hardness actually lives:

And keep the duo in frame. The sibling dive measures what the model does — indispensable, and structurally blind to why. This dive is the why. Behavioural science and anatomy, for the same organism; you want both, for the same reasons medicine does.

One paragraph of anatomy, inherited from dive three. Each token flows through the model as a single vector — the residual stream — and every attention head and feed-forward block reads from it and adds its contribution back in. Nothing overwrites; the stream accumulates. That design is the interpreter's luckiest break, because it means there is one shared bus where every component's output lands, in a common coordinate system, inspectable at every depth. The observables, then: the stream itself, the attention patterns (which earlier positions each head reads — you have already played with a real one, dive three's attention lens), and the final logits.

Now the instrument that best conveys what "reading the stream" feels like. The final layer of every model decodes the stream into next-token scores. The logit lens (nostalgebraist, 2020) is one impertinent question: what if we decoded earlier? Take the half-finished stream after layer 3, or 7, or 11, and push it through the same final decoder:

// the logit lens (nostalgebraist, 2020). the decoder was there the whole time.

for layer in model.layers:
    h = residual_stream_after(layer)   // the running vector, mid-flight
    logits = h @ W_unembed             // decode it as if we were already done
    print(layer, top_k(softmax(logits), 5))

// early layers: surface statistics. middle: the neighbourhood of the answer.
// late: the answer. the prediction is not born at the end — it FORMS.

Run that on a real model and something lovely appears: the prediction forms gradually. Early layers propose surface statistics; middle layers assemble the neighbourhood of the answer; late layers sharpen a decision the stream already mostly contains. Explore the shape of it:

The third prompt deserves a second look. The words are invented, so no fact from pretraining can be supplying "quell" — and the probability doesn't drift up, it jumps, early and hard. That is what it looks like when a small, specific mechanism switches on. The abyss names that mechanism (it was the field's first real circuit), and it was found by exactly this kind of poking. One honest caveat while we are being impressed: raw logit-lens readings are cleanest on some model families and noisier on others — refinements exist that fit a small learned decoder per layer — and "what the stream decodes to" is not the same as "what the layer is doing". Which is the cue for the second instrument.

Probing. Pick a property — part of speech, sentiment, "is this move legal", "is this statement true" — and train a small classifier on frozen activations to predict it. If a linear probe reads the property out of layer 6 with high accuracy, the model plausibly represents that property, linearly, by layer 6. Probes turned up something remarkable: models trained only to predict text carry crisp linear directions for a startling range of abstractions — position on a board, truth of a claim, identity of the speaker. But hold the instrument at arm's length as you read those results: a probe detects what is readable, not what is used. The training corpus correlates everything with everything; a sufficiently determined probe finds signals the model itself may never consult (the field runs control tasks — probes trained on nonsense labels — precisely to measure their own instrument's credulity). Correlation nominates. It never elects.

The load-bearing idea of modern interpretability is the linear representation hypothesis: the model's internal vocabulary is made of directions in activation space. You have met this before — dive two's geometry, where meanings had directions and arithmetic on them half-worked. A feature is a direction that consistently activates on some coherent property: this text is Python, this clause is negated, this person is being deceptive. Features, not neurons, are the proposed atoms of the computation.

Why not neurons? Because when you look, individual neurons are mostly polysemantic — one unit fires for academic citations and Korean text and the number 7. For years that read as noise. The reframe (Elhage and colleagues, 2022) is that it is compression: the model wants to track far more features than it has dimensions, and if features are sparse — each one rare, rarely co-occurring — it can pack them in at non-orthogonal angles, paying a little interference on the rare collisions in exchange for capacity. That packing is superposition, and it has a consequence you can watch: whether a model shares dimensions is a trained response to how often its concepts collide. So watch. This is not an animation — it is a real model, training on real gradients, in your browser, right now:

Do the experiment properly: leave it on dense and watch a triage — two dimensions, so the two most important features get them, orthogonally, and the rest are zeroed out of existence. Then switch to rarely co-occur and watch the same model, same dimensions, make the opposite choice: every arrow returns, crowding in at angles, and the reconstruction bench shows each feature reading back almost clean with a smear of contamination on its neighbours. Nothing in the widget's code places that geometry; the gradients do. Now scale the intuition up: a frontier model has thousands of dimensions per layer and millions of rare things worth knowing about text. Superposition is not a corner case. It is the operating regime — and it is why you cannot read a big model neuron by neuron: the features are real, and they are not axis-aligned.

So unpack them. A sparse autoencoder (SAE) is the move run in reverse (Bricken and colleagues, 2023): train a second, much wider network to re-express each activation vector as a sparse combination of directions from a large learned dictionary — tens of thousands of candidate directions, only a handful active at once. If superposition is how the model crams many features into few dimensions, the SAE rents the features a bigger space and asks each to stand in its own corner. And at production scale it works startlingly well (Templeton and colleagues, 2024): dictionaries trained on a production-scale model yielded millions of features with crisp, human-legible meanings — code-bug features, sycophancy features, and one famous cell that fired on mentions of the Golden Gate Bridge. Clamp that feature on — hold its activation high while the model runs — and the model steers every conversation back to the bridge, cheerfully, mid-recipe. Comic, and load-bearing: it demonstrated to a mass audience that a learned direction is not just a correlate. It is a lever.

Every instrument so far shares a weakness: it observes. The probe reads truth out of layer 6 — is the model using that direction, or is it a fossil of the training data that the computation routes around? Observation cannot say. The standard of proof is intervention: change the thing you claim matters, and watch whether behaviour changes the way your story requires. The workhorse is activation patching (the causal-tracing tradition — Meng and colleagues, 2022): run the model on a clean input and a corrupted twin, then transplant one internal activation from the clean run into the corrupted one and measure how much of the answer comes back.

// activation patching: the difference between "found" and "believed".

clean   = run("…Mary and John went to the bar, John handed a beer to")
corrupt = run("…Anna and John went to the bar, John handed a beer to")  // one name swapped

for site in every_head_and_layer:
    patched = run(corrupt.prompt, restoring=clean.activation_at(site))
    effect[site] = logit("Mary", patched) - logit("Mary", corrupt)

// a site where ONE transplanted activation restores the answer is in the circuit.
// correlation nominated the candidates; the intervention is what elects them.

The power of the method is its resolution: patch every head at every layer and you get a causal map — this head at this depth carries the name; this block turns it into the prediction. The abyss walks a real one. But notice what the method quietly requires: a hypothesis about where to patch and a metric for "the answer came back". Interventions are microscopes, not searchlights — they confirm mechanisms; they do not, by themselves, find them. Finding still belongs to the lenses, probes and dictionaries — which is why the loop is observe → hypothesise → intervene, never a single tool.

Now the honest catalogue of self-deceptions, because this field's maturity shows in how loudly it names them. Probes over-find — the control-task result from Level 2. Ablations under-prove: models are riddled with redundancy, so deleting a component and seeing nothing does not mean the component does nothing — a backup elsewhere may take over, and the abyss shows a circuit where exactly that happens. SAE features are a learned description, not the model's own ontology: the dictionary never reconstructs perfectly (whatever lives in the residue is computation your description simply misses), and the granularity is an artefact of dictionary size — train a wider one and "the Golden Gate feature" splits into finer sub-features, so "how many features does the model have?" has no dictionary-independent answer yet. And the scale wall: millions of features times billions of weights does not fit through human attention, so the field automates — models labelling another model's features — which buys coverage at the price of a circularity everyone names out loud: the microscope now contains the thing it is pointed at.