The model never sees your text. Two translations happen first: one by a frozen little program that was never taught what anything means, and one by a learned table where meaning is a location. Nearly every famous LLM blooper — from "count the r's in strawberry" to "why does Hindi cost me triple" — is an artifact of those two steps. (If "next-token predictor" is not second nature yet, start at the hub.) Stop wherever you like — each depth is complete on its own.
An LLM has never seen a letter. What reaches the network is a list of numbers, and getting there takes two steps that happen before anything you would call "the model" runs at all.
Step one: the tokenizer. A separate, frozen little program — not part of the neural network, not trained by gradient descent, not clever — chops your text into chunks drawn from a fixed vocabulary of tens to hundreds of thousands of entries, and replaces each chunk with its ID number. The capital of France is becomes something like [464, 3139, 286, 4881, 318].
Step two: the embedding table. The first layer of the actual network swaps each ID for a learned vector — a long list of numbers, a few thousand of them. That vector, not your text and not the ID, is what every subsequent computation operates on.
Letters, words, spelling, spaces — surely the machine sees what is on your screen. It does not. There is no point in the pipeline where a character exists as a thing the model can inspect. Asking it about the letters in a word is like asking you about the grain of the paper a book was printed on: you read the book, not the fibres.
Your sentence arrives as a sequence of dictionary-entry IDs — chunks from a fixed phrasebook compiled once, before training, by a compression algorithm. Common words are a single entry. Rare words are glued together out of fragments. And whatever is inside an entry is invisible: the chunk is the smallest thing that exists.
Here is the fact that everything else in this guide hangs off: the phrasebook was optimised for compression, not for meaning. When strawberry splits into straw + berry, it splits there because corpus frequency said so. Botany was not consulted. Nobody checked whether the pieces were morphemes, syllables, or nonsense. The only question the algorithm asked was: which pair of things turns up next to each other most often?
Because those chunks are the atoms of the model's world, every quirk of the compression scheme becomes a permanent quirk of the model. It can reason about an atom, combine atoms, predict the next atom — but it cannot look inside one, any more than you can look inside a pixel. Readers of the hub met the consequence there: counting the r's in "strawberry". This guide is the mechanism.
One more thing worth taking at the surface, because it explains a lot later. The tokenizer is the only piece of the stack that is trained separately and then frozen. It was "trained" in a sense — statistically, on a corpus — but by counting, not by learning. And once the model's own training begins, the phrasebook can never change again: every weight in the network is fitted to exactly those atoms. A tokenizer and the model on top of it are two systems, welded together on day one, that can nevertheless disagree about the world.
An LLM reads dictionary entries, not letters. The dictionary was built by a compression algorithm that knows nothing about language, and every quirk that compressor has, the model inherits — permanently.
Start with the design question, because the answer is not obvious. You need to turn text into a bounded list of symbols. What should a symbol be?
Characters, or raw bytes. A tiny vocabulary, and nothing is ever unrepresentable. But sequences get several times longer, and length is the expensive dimension: every token costs one forward pass, and attention makes each token look at everything before it. A lone byte also carries almost no signal — e tells the network nearly nothing on its own.
Whole words. Short sequences, and each symbol is meaningful. But the vocabulary explodes — names, typos, code identifiers, and morphology-rich languages generate words without limit — and the world is open: there will always be words the table has never seen. A fixed table has no row for tomorrow's product name.
Subwords — the compromise that won. Let frequency decide. Strings common enough to be worth a slot get their own entry; everything else decomposes into reusable pieces. No linguist required, no unknown-word problem, and sequence length stays close to the whole-word case for text that looks like the training corpus.
The algorithm that does this is byte-pair encoding — originally a 1994 data-compression trick (Gage), repurposed for machine translation vocabularies in 2016 (Sennrich, Haddow & Birch) and taken to internet scale by GPT-2 in 2019, which ran it over raw bytes so that no string on earth is unrepresentable. It is honestly simple:
vocab = set(all 256 byte values) // so nothing is ever unrepresentable merges = [] // an ORDERED list. the order is the whole point while len(vocab) < target: pairs = count_adjacent(corpus) // every neighbouring pair, weighted by frequency best = argmax(pairs) // e.g. ("t", "h") — the commonest pair in the corpus if pairs[best] < 2: break // nothing repeats any more: you are done corpus = replace_everywhere(corpus, best, best[0] + best[1]) vocab.add(best[0] + best[1]) merges.append(best) // rule N: "seeing t then h? glue them." // the merge list IS the tokenizer. no grammar, no dictionary, no model — // just these rules, in this order, on every string, forever.
At inference there is no statistics left at all. You take the learned merges and apply them to your text in priority order — rule 1 wherever it fits, then rule 2, and so on. Which makes tokenization completely deterministic: same text, same tokens, every single time. Of all the stages between your keyboard and a reply, this is the one with no randomness anywhere in it.
Reading that is one thing. Below, it runs. Tab one trains a real BPE on a few kilobytes of English — press Merge the top pair half a dozen times and watch the vocabulary come into existence. Tab two hands you the tokenizer you just grew.
Step through the first merges slowly and you see the algorithm's whole character. The opening rules are not words and not morphemes: a space glued to a letter, then he, then those two combined into ·the. Nothing in there knows what a word is. It is counting, and counting alone, and published merge lists from real tokenizers open with the same unglamorous kind of entry.
Four things the lab shows that you will meet again in production:
It has never heard of words. It is pair-frequency counting and nothing else. When unhappy comes apart into un + happy it looks like morphology — but the algorithm found un frequent, not meaningful, and it would have merged xq just as happily had the corpus contained enough of it. The linguistics is a coincidence of English being made of reusable parts; the compression is the point. (Other families count slightly differently — WordPiece scores candidates by likelihood gain rather than raw frequency, and SentencePiece's unigram model prunes a large vocabulary downward instead of growing one up. Same compromise, different arithmetic. Not worth a tour.)
The knob. Make the vocabulary bigger and sequences get shorter: cheaper compute per document, more content inside a fixed context window. But the embedding table grows with it — at 128,000 entries by a few thousand dimensions you are carrying hundreds of millions of parameters in the lookup table alone, and you pay for it a second time at the output layer, which has to produce a score for every entry on every forward pass. Worse, the rarest entries in a large vocabulary barely occur in training, so their vectors stay close to wherever random initialisation put them. That is a loaded gun, and Level 5's first pod is where it goes off.
The industry has converged on a band rather than an answer: roughly 50,000 to 250,000 entries. GPT-2 and GPT-3 used 50,257; more recent families sit somewhere in the 128,000–200,000 range. There is no principled optimum here, only a trade curve that different labs price differently.
The multilingual bill. This is the part with a victim, so give it a moment. A tokenizer trained on an English-heavy corpus compresses English best — that is not a bias someone inserted, it is arithmetic: merges are allocated to whatever was frequent, and what was frequent was English.
The metric is fertility: tokens per word. English runs around 1.3. Languages distant from the training mix, or written in scripts the merge list barely saw, can run two to five times that, and the worst cases fragment most of the way back to raw bytes — which is exactly what you can watch happen in the lab above, where a sentence with an accented character falls apart into individual bytes because no merge rule ever covered it. Same meaning, several times the tokens. (Petrov and colleagues measured this across a large set of languages in 2023 and put numbers on the disparity; the finding has held up as a general property of frequency-trained vocabularies.)
And the bill arrives three ways at once, which is what makes it more than a curiosity:
None of that is the model being stupid about the language. It is a design consequence baked in before training started, invisible unless you actually count tokens — which is why the language presets in the lab are worth clicking twice.
The failure zoo. Every exhibit below is famous, and every one has the same root cause: character-level structure does not exist inside the model's atoms. What differs is the mechanism by which the blindness shows up.
The model is not bad at spelling — it is blind to it, and no prompt fixes a missing sense. Character-level work belongs in your code, not in the prompt: count, reverse, slice and validate in a tool, and let the model do what its atoms allow. Readers who reach the hub's ecosystem pod will recognise this as one half of why tool use exists at all.
The embedding table is a matrix: one row per vocabulary entry, a few thousand columns wide. Row i is token i — there is no other definition of it anywhere in the network. Look up the row, and that vector is what flows into the first transformer layer.
At the start of training, that matrix is random noise. Nobody inserts meaning into it and there is no stage where a human labels anything. Meaning condenses under prediction pressure: to predict text well the network needs to generalise, and the cheapest way to reuse what it learned about dog when it meets puppy is to have moved their vectors close together. Every gradient step that helps one word by nudging another is a step that pulls similar-context words into similar positions. Nothing is arranging the space; the space is what falls out.
This is the distributional hypothesis made mechanical. Firth put it in 1957 as "you shall know a word by the company it keeps" — a claim about linguistics. Gradient descent turns it into an engineering fact: usage similarity is the only signal available, so usage similarity is what the geometry encodes.
And geometry is something you can compute with. Three consequences, in increasing order of surprise. Closeness — measured as cosine similarity, the angle between two vectors — tracks similarity of usage. Clusters form on their own: animals here, capital cities there, error messages somewhere else, with nobody drawing the boundaries. And some directions turn out to encode relations, which is the finding that made this famous: king − man + woman lands near queen (word2vec, Mikolov et al., 2013; GloVe, Pennington et al., 2014, reproduced it with different maths). Say the caveat in the same breath: the celebrated analogies are the showcase cases. The geometry is real; the arithmetic party trick is brittle outside the examples it made its name on.
def similar(a, b): // a and b are token vectors: a few thousand floats each return dot(a, b) / (norm(a) * norm(b)) // cosine: the ANGLE between them, length ignored def nearest(word, k = 5): v = E[token_id(word)] // one row of the embedding table — that row IS the token scored = [(similar(v, E[i]), i) for i in range(len(E))] return top_k(scored, k) // that second function is the heart of every semantic-search product you have // used. production systems index it so it is not a full scan — but the thing // being approximated is exactly this loop.
The map below is illustrative in its layout and real in its arithmetic — click a word and the five nearest are ranked live off the coordinates you can see; switch to Analogy and the vector subtraction is done in front of you.
Nearby means used in the same contexts — and that is exactly where antonyms live. hot and cold, buy and sell, always and never: near-identical company, opposite meaning, neighbouring vectors. Click hot on the map above and read what comes back second.
The consequence for anyone shipping semantic search is concrete and expensive: similarity is topical, not logical. "Flights under $100" and "flights over $100" embed almost identically, and no threshold on cosine similarity will ever separate them, because the difference between them is not a difference in what they are about. Negation, quantities, dates, permissions and units belong in filters and code. The vectors find the neighbourhood; they will not tell you the truth once you are in it.
Static vectors are only the starting point. The row you look up is the same row every time — bank enters "the river bank" and "the bank deposit" as one identical vector. It does not stay identical. Inside the transformer that vector is rewritten layer by layer using the surrounding context, and the two banks come out as two different things. The table supplies a word's prior; context supplies the word. The machinery that does the rewriting is attention — the next dive in this series.
Standalone embedding models deserve a paragraph, because you will be asked to use one. Some models exist only to emit a vector — not a next token, but one fixed-length vector for a whole sentence or document, trained so that texts meaning similar things land close together. That single trick powers semantic search, deduplication, clustering, recommendation, and the retrieval half of RAG — where this paragraph becomes a whole machine, in the RAG dive. Production dimensions run from a few hundred to a few thousand, and the engineering question is almost always storage and recall speed rather than model choice.
Tokenization is bookkeeping; embeddings are where meaning lives — as position in a learned space. Nobody drew the map. Prediction pressure did, and it drew it according to usage, which is close enough to meaning to be useful and far enough from it to be dangerous.
In February 2023 two researchers — Jessica Rumbelow and Matthew Watkins — went looking for the tokens with the strangest embeddings in GPT-family models and found a small set that broke them. Asked to repeat one back, the models would insult the user, change the subject, emit unrelated words, or claim they could not see anything at all. The most famous of them was SolidGoldMagikarp.
The mechanism is the cleanest possible illustration of Level 1's warning that the tokenizer and the model are two systems. These strings were Reddit usernames that appeared thousands of times in the corpus the tokenizer was trained on — one of them from a subreddit dedicated to counting, where the same handful of accounts posted incessantly. Frequency is all BPE looks at, so they earned vocabulary entries of their own. Then the model's training corpus was filtered differently, and those entries almost never appeared in it.
So the rows existed, and nothing ever trained them. Their vectors stayed near wherever initialisation dropped them: a location in the space that carries no learned relationship to anything. Feeding one in hands the network a vector from nowhere and asks it to continue — and the behaviour that comes back is whatever the untrained region happens to decode to.
This is Level 3's undertraining risk made flesh, and a lovely proof that the join is real: two components, two corpora, one disagreement, and a model that starts speaking in tongues.
"Byte-level" sounds like a technicality until the world stops being ASCII. An emoji is several bytes, so unless the merge list happened to fuse them it is several tokens. A family emoji is multiple emoji joined by zero-width joiner characters — every joiner a byte sequence, every byte sequence a potential token — which is why one glyph can cost a dozen. Accented characters that were composed from a base letter plus a combining mark split at the seam, and the same visible letter can tokenize two different ways depending on which Unicode normalisation produced it.
Then there is the sharp edge. Invisible characters are real tokens. Zero-width spaces, zero-width joiners, bidirectional overrides, variation selectors: they survive copy-paste, they are not visible in any renderer, and they consume vocabulary slots and context budget like anything else. A block of text that looks like two sentences can carry an arbitrary amount of content a human reviewer will never see.
That is a security property, not just a curiosity — and it is the same one-channel problem the hub's prompt-injection section covers: everything reaching the model is tokens, and the model has no notion of which of them a human could see. Sanitise and normalise text before it enters a context window, for the same reason you escape it before it enters a database.
The tokenizer is chosen and frozen before pretraining, and it can never be swapped without retraining the model. Every embedding row is fitted to one specific set of atoms, and every circuit built on top of those rows assumes them. Change the chopping and the entire network is reading a language it has never seen.
Three consequences engineers actually hit:
Per-token prices are not comparable across vendors. Two providers can quote the same price per million tokens and charge you meaningfully different amounts for the same document, because their tokenizers disagree about how many tokens the document is. Normalise per character, or per representative request, before you compare anything.
Token counts do not transfer. Counting your prompt with one family's tokenizer library tells you nothing reliable about another family's bill or context usage. If you budget context, budget it with the tokenizer of the model you are actually calling.
"How many tokens is this?" has no model-independent answer. The ~4-characters-per-token rule of thumb is a useful envelope for English and a poor one for everything else. It is a planning heuristic, never an accounting figure.
There is a research line that regards the phrasebook as the original sin, and its argument is strong: every failure in Level 3 exists because a compression algorithm chose the atoms. Remove the tokenizer and the strawberry problem, the digit problem, the multilingual inequity and the glitch tokens all disappear at once, because none of them are properties of neural networks — they are properties of a fixed vocabulary.
The straightforward version is to operate on characters or bytes directly. ByT5 (Xue et al., 2021) did exactly that and demonstrated the trade honestly: markedly better robustness to noise, misspellings and unusual text, paid for with sequences several times longer. Newer byte-latent approaches try to have it both ways by learning to group bytes into dynamic patches — segmentation that adapts to the text rather than being fixed in a table before training.
Why tokenizers survive anyway: compression is compute economics. Attention cost grows faster than linearly in sequence length, and a three-to-four-times longer sequence is a three-to-four-times bigger bill on every single request, forever. That is an enormous amount of budget to spend on correctness that most workloads never notice. Frame it as an open trade, not a settled verdict — the tokenizer persists because it is cheap, not because anyone thinks it is right.
The space is anisotropic. Real embeddings do not fill their space evenly — the vectors crowd into a narrow cone, so almost every pair of unrelated words still scores a fairly high cosine similarity. The practitioner's takeaway: an absolute similarity number is close to meaningless. A 0.82 is not "82% related"; it may be below average for your corpus. Only relative comparisons — this candidate against that one, ranked — carry information, which is why a fixed similarity threshold is one of the most common ways to ship a broken retrieval system.
The geometry aligns across languages far better than it has any right to. Train on many languages and the same concept ends up in a similar constellation regardless of the script it was written in, because the contexts a concept appears in are similar everywhere. The practitioner's takeaway: multilingual retrieval often works cross-lingually out of the box — a query in one language can find documents in another without translating anything — and it is worth testing before you build a translation layer. (This is also the phenomenon behind the language-independent features the hub's interpretability pod describes.)
Matryoshka embeddings are trained so that the first N dimensions are themselves a usable, lower-quality embedding — nesting a 256-dimensional vector inside a 1,536-dimensional one. The practitioner's takeaway: you can truncate the vector to trade accuracy for storage and search speed, and a common pattern is to retrieve broadly on the short prefix and re-rank the survivors on the full vector. Check whether the model you are using was trained this way before you truncate; on one that was not, chopping dimensions is just damage.