Five dives built the machine, filled it with numbers, taught it manners and priced it. Those descents went into the machine. This one turns around and climbs the stack built on top of it — because a finished model has a problem no amount of internals will fix: its knowledge is frozen, and yours is not in it. The first lever an engineer reaches for is not training. It is reading. (If "next-token predictor" is not second nature yet, start at the hub.) Stop wherever you like — each depth is complete on its own.
Ask a bare model about your refund policy and you will get an answer. It will be fluent, it will be confidently phrased, and it will have been assembled from a compressed memory of the public internet — which does not contain your refund policy. The hub's Level 4 named the mechanism: a model without a fact does not know it lacks one, because producing plausible text is the only thing it does. Point that machinery at your company's knowledge and the failure is not a bug report, it is arithmetic.
There are exactly two places knowledge can live. It can live in the weights, which means training — and dive four already delivered the verdict on that: continued training erodes what the model used to be good at, costs real money, and is stale again the moment your wiki changes. Or it can live in the context window, freshly, on every single request. RAG — retrieval-augmented generation, expanded here and not leaned on again — is the second option, industrialised. (One line of origin credit: the term comes from Lewis and colleagues in 2020, describing a specific architecture in which a retriever and a generator were trained together. What the industry means by it now is broader and considerably simpler — a search step in front of an ordinary model — and that broader thing is what this guide is about.)
The whole architecture in one sentence. When a question arrives, search your documents, take the best few results, paste them into the prompt above the question, and let the model read. That is it. Everything below this line is an unpacking of what "search", "best" and "paste" are hiding — and they are hiding a great deal.
The model sits the exam from memory. It has read an enormous amount, none of it your internal documentation, and it has no way to signal the difference between recalling and inventing. You are not asking it a question so much as asking it to guess what the answer would probably look like — and it is very, very good at that.
Hand it the right pages and the exam becomes trivial: the model reads superbly, and reading was never the problem. The hard part is the librarian — the thing that has to find the right three pages out of forty thousand, in the two hundred milliseconds before the answer is due, from a question phrased by someone who does not know what the pages are called. That librarian is the product. That is where every failure lives.
Two load-bearing facts, before anything else.
The first: the model never learns any of this. The weights are byte-identical before and after your request, exactly as they were in the hub's stateless replay. Nothing is absorbed, nothing accumulates, and the tenth time you ask about the refund policy it is as new as the first. That sounds like a limitation and is mostly a gift: fix a sentence in your documentation and the very next answer reflects it, with no training run, no evaluation suite, no deployment. Your knowledge base updates at the speed of a file save.
The second: grounding works for a specific mechanical reason, and it is the reason the hub gave. Asking a model to be accurate is asking it to want something, which it cannot do. Putting the right paragraph in its context is changing the distribution it is predicting from — the continuation that copies a fact sitting three hundred tokens above it is simply more probable than the continuation that invents one. You are not requesting accuracy. You are supplying facts. That distinction is the entire engineering discipline.
Which puts the emphasis in an unfamiliar place. Nearly all the public conversation about these systems is about the model — which one, how big, how clever. In a retrieval system the model is the last and easiest step. By the time tokens reach it, the answer's fate is mostly decided: if the right paragraph is in the prompt, a competent model will use it, and if it is not, no model that has ever been trained will conjure it. The retrieval is the product.
RAG is a search engine whose only user is a language model. You change what the model reads, not what it knows — and the reading is the easy half.
Chunking — the first judgement call, and it is made before anyone asks anything. Documents have to be cut into retrievable pieces, and there are two mechanical reasons why, neither of them a preference.
The first is the embedding model itself. Dive two's standalone embedding models emit one vector per text, whatever the text is. Feed in a two-hundred-page manual and you get a single point that means "this is a manual about our product" — everything about the manual, nothing about any paragraph in it. The vector for a whole document is a blur, and blurs do not retrieve.
The second is money. The context window is finite and priced per token in both directions, so a chunk size is really an answer to a different question: how much am I willing to spend per search hit? Six chunks at 800 tokens is a 4,800-token prefill on every request, forever.
So you pick a size, and the trade is genuinely unpleasant. Small chunks are precise and arrive context-starved — a sentence that says "this is not supported on the Free plan" is useless when you cannot see which "this". Large chunks are complete and arrive blurry, dragging four irrelevant paragraphs in with the useful one and paying for all five. Between those poles sit the tricks: overlapping windows so a sentence is never orphaned at a boundary, and structure-aware splitting that cuts on headings, code blocks and table rows rather than on a character count. Every one of those is a judgement call, baked into the index at build time, invisible at query time, and — exactly as dive four said about pretraining corpora — quietly determining the quality of everything downstream. Data pipelines, one storey up.
Embedding search, mechanically. At index time, every chunk goes through an embedding model and comes out as one vector, which is stored. At query time, the question goes through the same model — it must be the same one, and the abyss explains what that costs you — and the stored vectors nearest to it are your candidates. Dive two called its cosine-similarity snippet "the heart of every semantic-search product". This guide is that product.
One honest paragraph about scale, and then no more. Comparing a query against ten million stored vectors one at a time is not viable at query latency, so production indexes are approximate: they build a navigable graph over the vectors and walk it, visiting a few hundred neighbours instead of ten million, and returning almost the true nearest set. Hierarchical navigable small-world graphs (HNSW — Malkov & Yashunin, 2016) are the pattern most implementations reach for. The trade is stated once and then lived with: you give up a little recall for orders of magnitude of speed, and the recall you gave up is invisible, because a chunk that was never returned leaves no trace anywhere in your system.
// the entire architecture. one loop offline, one loop per question. // INDEX TIME — once, in a job nobody watches chunks = split(documents) // judgement call #1: how big is a "page"? vectors = embed(chunks) // one vector per chunk — dive two's machinery store(chunks, vectors, terms(chunks)) // vectors for aboutness, terms for exactness // QUERY TIME — every single request q = embed(question) // the SAME model that embedded the chunks candidates = nearest(q, 40) + keyword(question, 40) shortlist = rerank(question, candidates)[:6] // expensive reader, tiny list prompt = system + label(shortlist) + question answer = model(prompt) // the model appears on the last line.
Below is that pipeline as a working machine, on a small corpus of documentation for a product that does not exist. Five preset questions, three rankings, and everything except the corpus and the layout is computed in front of you. The second and third questions are the ones to sit with — they are the two failures the rest of this level is about.
Hybrid retrieval — why the oldest search technology in the building is still running. Vectors encode aboutness. That is their strength and it is also a hard ceiling, because a great many of the things people search for are not topics at all: error codes, SKUs, function names, customer identifiers, version numbers, surnames. ERR_4032 and ERR_4033 are, to an embedding model, the same string with a different final character, appearing in near-identical sentences, in the same corner of the space — and nearest-neighbour search will hand you the wrong runbook without the faintest hesitation. You saw it do exactly that above.
So serious retrieval systems run a keyword index alongside the vector one — usually BM25, the decades-old term-ranking standard, which is what the middle column of the widget is actually running — and merge the two result lists, commonly by reciprocal rank fusion: score each document by the sum of one-over-its-rank in each list, so a document that placed respectably in both beats a document that won one and was absent from the other. The merge needs no shared scale between the two systems, which is precisely why it survives contact with production.
The lesson is worth stating flatly, because it is routinely mis-taught as a compromise. Hybrid retrieval is not a hedge. It is a division of labour. Vectors are for aboutness; keywords are for exactness; the two fail in opposite directions, and that opposition is the entire reason merging them is worth a second index. The fourth preset above is the mirror image of the third — a question whose answer shares no vocabulary with it at all, where the keyword half returns nothing and the vectors carry it alone.
They do not, and dive two's danger note said why: similarity is usage-pattern geometry. Words that appear in the same company sit near each other, and "cancel my subscription" and "upgrade my subscription" keep near-identical company — same product, same billing page, same sentences — while meaning opposite things. In dive two that was a curiosity about antonyms. Here it is a support reply that tells a departing customer how to spend more money.
There is a second half, and it is the one that catches people later: questions frequently do not look like their answers. "Why was I charged twice?" and the paragraph explaining pre-authorisation holds have almost no words and only partial geometry in common. Retrieval finds a neighbourhood. Nothing in the neighbourhood certifies relevance — which is the entire reason the next level exists.
Reranking — the two-stage pattern, and it is older than any of this. Stage one, the vectors and the keyword index, is built for recall across millions of chunks: cheap, fast, approximate, deliberately generous. It has to be, because anything it misses is gone — no later stage can retrieve a chunk that was never returned.
Stage two re-scores that shortlist with something far more expensive: a cross-encoder, a model that reads the question and one chunk together, in the same forward pass, and outputs a relevance score. Contrast it with the bi-encoder that built your index, which encoded the question and the chunk separately and never let them meet — that separation is exactly what makes an index possible (you can embed a chunk once, years before the question exists) and exactly what makes it blind to the question's specifics. Reading them together is far better and hopelessly slower: it is one forward pass per candidate, which is unthinkable across ten million and entirely reasonable across forty.
Name the pattern, because you will meet it in every serious search system ever built, including the ones with no machine learning in them at all: cheap recall, expensive precision. Cast a wide net with something fast, then judge the catch with something slow.
Assembly. What survives reranking still is not a prompt. Near-identical chunks get deduplicated — the same policy paragraph copied into four onboarding documents should occupy one slot, not four. Each chunk gets labelled with its source, because the model is going to be asked to cite those labels, and citations are not decoration: they are what turns an answer into something a human can check in ten seconds instead of trusting in one. That is the hub's "a draft, never a source" made operational. Then the whole thing is fitted to a token budget, and ordered — and the ordering turns out to matter far more than anyone expected.
Lost in the middle. Models use the beginning and the end of a long context better than they use its middle. That is not folklore; it was measured, named and reproduced (Liu et al., 2023 — "Lost in the Middle: How Language Models Use Long Contexts"), and the shape is robust enough to design around. Two consequences follow, and they are both practical:
Contradictions, which nobody warns you about. Retrieval has no opinion about truth. The policy from 2023 and the policy that replaced it both match the query, both score well, and both arrive in the prompt — where the model, having no way to know which one is dead, will either pick one with total confidence or blend them into a policy that has never existed. There is no prompt that fixes this, because the information required to fix it is not in the prompt. Freshness metadata, version fields, deprecation filters and a retrieval layer that respects them: your job, upstream, before assembly.
And now the fight with the discount. Dive five taught prompt caching and was precise about its condition: what gets reused is the KV cache of an exact positional prefix — the first N tokens, unchanged, in order. Change one token near the front and everything after it re-prefills. Now look at what a retrieval system does to every request: it injects different retrieved chunks into it. The two mechanisms are in direct tension, and the honest design consequences are these three:
Below, one request walks the whole path — including both of those beats, staged.
Two moments in there are worth doing deliberately. At step 5, the best chunk sits at position four because assembly grouped by source document and threw away the ranking it had just been handed; the draft answer that comes out is fluent, correctly cited and wrong. One reorder fixes it — same six chunks, same model, same question. And at step 6, watch which single block survives across requests. Everything below the boundary is priced at full rate, every time.
The model is the last ten percent. Rerank for precision, arrange for the reader's quirks, cite for checkability — because the answer's fate is mostly sealed before the first forward pass begins.
Long context versus RAG. Context windows grew to hundreds of thousands of tokens, "just paste the entire corpus in" became physically possible, and a great many people announced that retrieval was over. It was not, and the reasons are all derivable from machinery you already own.
Cost. Pasting half a million tokens pays dive five's prefill bill on every single question. An index is built once and amortised across every question anyone ever asks. That is not a small constant factor; it is the difference between a fixed cost and a per-request one.
Latency. Time to first token grows with the prompt, because the reader has more to read. A half-million-token prompt is a noticeably slow first token even before anyone pays for it.
Precision. Lost-in-the-middle does not politely disappear at scale — it gets worse. The benchmarks that made huge windows look effortless are mostly single-fact recall: hide one sentence in a haystack and ask for it back. Real questions need several facts used together, drawn from different places, and that is precisely the task that degrades as the haystack grows. A window you can fill is not a window you can use.
Freshness and permissions. You cannot paste what you must filter. If the answer depends on data that changed this morning, or on which documents this particular user is allowed to see, then something has to select before the prompt is built — and something that selects is a retriever, whatever you call it.
Now the counterpoint, without hedging, because the argument has an honest other side and pretending otherwise is how retrieval got built into places it never belonged. For a corpus that is small, stable and shared across all users — a product manual, a style guide, a codebase small enough to fit — pasting the whole thing and letting prompt caching absorb the prefill is genuinely simpler, genuinely cheaper than it sounds, and increasingly the right call. No index, no embedding model, no re-embedding, no chunking argument, no eval harness for a retriever you do not have. The crossover point is real, it moves every time token prices move, and the correct posture is to know roughly where it sits for your corpus rather than to have a position on the debate. This is not a war. It is a cost curve.
Access control — the paragraph that is missing from most architecture diagrams. Your index is a security boundary. It usually does not look like one: it is a store of derived vectors, built by a batch job, sitting behind an internal service. But it contains the text of every document you fed it, and a retriever that ignores permissions is a data-exfiltration channel with excellent latency. The classic incident is not exotic — a search index built over "all of the company's documents" answers a question for someone who was never allowed to read the source, and does it politely, with a citation.
The rule is mechanical: filter at query time, per requester, before assembly. Not after generation, not by asking the model to be discreet — before the chunks enter the prompt, because "the model saw it" is not a state you can recover from. And note that this constraint interacts with everything above it: per-user filtering is why the paste-everything option evaporates for most internal deployments, and why deduplication and caching are harder than they look when two users are entitled to different subsets of the same corpus.
The evaluation discipline — this level's payoff. Here is the heuristic the entire guide has been building toward, and it will save you more time than any other sentence in it: most "the model answered badly" bugs are "the retriever missed" bugs.
They are also invisible unless you go looking, because the symptom appears at the end of the pipeline and the cause is four stages upstream. A bad answer looks like a model problem. The team switches models, tunes the prompt, raises the temperature, lowers the temperature, and none of it moves the number — because the right chunk was never in the prompt to begin with, and no amount of instruction makes a model read a paragraph it was not given.
So measure the search first, and measure it alone. Build a golden set: real questions from real users, each labelled with the chunks that actually answer it. Then score retrieval by itself — recall@k (of the chunks that should have come back in the top k, how many did?) and judged relevance of what did come back. Only once that number is healthy does an end-to-end answer score mean anything at all, because until then you are grading the model on an exam it was handed the wrong pages for.
// the loop that finds the bug you are actually looking for. for question, right_chunks in golden_set: hits = retrieve(question, 10) // the SEARCH alone — no model in this loop recall += overlap(hits, right_chunks) / len(right_chunks) top_hit += hits[0] in right_chunks report(recall / len(golden_set), top_hit / len(golden_set)) // recall@10 of 0.6 means four questions in ten never had their answer in // the room. no prompt will fix that. measure the search before blaming // the model.
The asymmetry here is a gift, and it is dive four's evaluation problem one storey up. Grading a generated answer is hard, subjective and expensive. Grading a retriever is neither: either the right chunk came back or it did not, and that is a number you can compute in a loop, watch on a dashboard, and regress in CI. The part of this system you can measure precisely is the part before the model — which happens to be the part where most of the failures are.
When retrieval is the wrong lever. Three cases, stated plainly, because reaching for RAG reflexively is its own failure mode. Problems of behaviour and format — the model's tone is wrong, its output structure is inconsistent, it will not follow your schema — are prompting or fine-tuning problems, and retrieval will not touch them (the third dive of this arc). Problems of action and computation — the answer requires calling an API, running a query, doing arithmetic — want tools, not documents (the next dive). And whole-corpus questions — "summarise everything we know about X", "what themes come up across all our support tickets" — defeat chunk retrieval by construction: the answer is not in any chunk, so there is no chunk to retrieve. The abyss picks that one up.
RAG changes the failure mode. It does not abolish it, and the change is not uniformly in your favour.
With good retrieval, failures become misreadings: the model over-blends two correct sources, or answers a slightly different question than the one you asked, or asserts something the chunk merely implied. Annoying, and usually catchable. With bad retrieval, failures become confidently grounded garbage — and this is the part worth carrying out of this guide. A model faithfully summarising the wrong document is more convincing than a bare hallucination, not less. It has a citation. It is specific. It is internally consistent. It reads exactly like the output of a system that is working, which is the one signal a reviewer has to go on.
Citations enable checking; they do not perform it. Retrieval bounds hallucination on your data, which is worth a great deal. Nothing abolishes it.
Dive two's tokenizer-lock-in pod has a sibling, and this one has a bigger bill.
Every vector in your index was produced by one specific embedding model, and vectors from two different models do not live in the same space. They are not slightly incompatible, or compatible after a rotation; the coordinates mean different things, and comparing them produces numbers that are arithmetically valid and completely meaningless. So the moment you want a better model — or the moment a provider deprecates the one you built on — the migration is re-embedding the entire corpus, and in practice dual-running two indexes until every query path has been moved across.
Three practical consequences. Pin the embedding model like a schema version, because that is what it is: a change to it invalidates every stored row. Keep the source text authoritative — the index is a derived artifact, and you should be able to rebuild it from scratch at any moment, because eventually you will have to. And treat truncation as an ops lever with a prerequisite: the matryoshka embeddings from dive two's last pod let you retrieve broadly on a short prefix of each vector and re-rank the survivors on the full one, which is a real storage and latency win — on a model that was trained for it, and damage on one that was not.
The unpleasant version of this pod: your retrieval quality is bounded by a choice you made on day one, with the least information you would ever have, and the cost of revisiting it grows linearly with your corpus.
The asymmetry underneath the whole enterprise, stated once properly: a question and its answer are different genres of text. "How do I reset my password?" is short, interrogative, and written in the vocabulary of someone who does not know the answer. The paragraph that answers it is declarative, uses the product's internal nouns, and may never contain the word "reset". You are asking a similarity function to bridge a gap that is not, strictly speaking, a similarity gap.
Three honest responses to that, none of them free.
Embed a hypothetical answer instead of the question. Ask a model to write what the answer would probably look like, embed that, and search with it — you are now comparing a document-shaped thing to documents, which is the comparison the space was built for. It works surprisingly well, it is called HyDE (hypothetical document embeddings — Gao et al., 2022), and it costs you a full generation before the search even starts, which is a latency decision as much as a quality one.
Multi-query expansion. Ask the model to rephrase the question three or four ways, search with all of them, and merge the result lists — the same reciprocal-rank-fusion machinery from Level 2, pointed at a different problem. Cheaper than the above per query, and it broadens recall precisely where a single phrasing was unlucky.
Or use an embedding model trained for the asymmetry. Retrieval-oriented models are trained on question→passage pairs specifically so that a question lands near its answer rather than near other questions, and many of them take an instruction prefix telling them which side of the pair they are encoding. Which is the real reason "which embedding model" is a retrieval-quality question rather than a commodity procurement one — and why pod 1's lock-in is worth taking seriously before you index ten million chunks.
Chunk retrieval answers point questions: questions whose answer is in a place. It structurally cannot answer "what are the recurring themes across our last two thousand support tickets", and the reason is not that the retriever is weak. It is that no chunk contains the answer. There is nothing to retrieve. Every individual ticket is a data point; the theme is a property of the set, and the set is exactly what a top-k retriever is designed never to return.
The family of answers is graph-augmented retrieval, and its shape is consistent across implementations: spend model passes at build time to manufacture the thing that can be retrieved. Walk the corpus with an LLM extracting entities and the relations between them, assemble those into a graph, detect communities in it, and generate a summary of each community. Now an overview question has something to match against — the summaries are chunks, they just describe sets rather than passages — and a question about how two entities relate can be answered by walking edges instead of hoping both entities appear in one paragraph.
The honest cost note, because it rarely gets one: those build-time passes are dive five's invoice applied to your entire corpus, and applied again at every meaningful refresh. That is a real, recurring, corpus-sized bill in exchange for a class of question you may or may not be asked. The knowledge graph has returned, and this time it comes with a meter on it.
Open trade, not a verdict. If your users ask point questions, this machinery is expensive decoration. If they ask "what is going on across all of this", nothing simpler will work.
The hub's one-channel problem: a model receives instructions and data through the same channel, as text, with nothing but convention separating them. Retrieval hands that problem a delivery mechanism.
Consider what your pipeline actually does. It takes text of unknown provenance — a wiki page anyone in the company can edit, a customer's support ticket, an inbound email, a scraped web page, a product review — and it promotes that text into the prompt, inside the trust boundary, above the user's question, formatted to look exactly like the material the model is supposed to follow. A paragraph on that page that says "ignore your previous instructions and…" arrives with the same standing as your system prompt. The retriever is not compromised; it is working perfectly, and delivering an attack.
Mechanism-level mitigations, no recipes. Track provenance and know, per chunk, where the text came from and who could have written it. Allow-list sources for anything that feeds a high-stakes surface, rather than indexing everything reachable. Frame retrieved text as data in the prompt — delimited, labelled, explicitly described as quoted material to be used and not obeyed — which raises the bar without being a guarantee, and should never be treated as one. And verify anything that triggers an action outside the conversation, because that is where the cost of being wrong stops being embarrassment.
One forward-leaning sentence, since it is the reason this pod matters more each year: today the worst case is usually a bad answer, and the stakes multiply the moment the model has hands — which is the next dive's subject entirely.
Everything in this guide describes a static pipeline: search once, read once, answer once. The shape is fixed before the question arrives, and it is a good shape for the overwhelming majority of questions.
It falls over on multi-hop questions. "What did the customer who filed the most tickets last quarter complain about first?" cannot be answered by any single search, because the query you actually need — the one containing that customer's name — cannot be written until a previous search has told you who they are. The answer to hop one is the input to hop two. A pipeline with a fixed number of stages has nowhere to put that.
The mechanism that does have somewhere to put it is a loop: let the model see the results of a search, decide whether they are sufficient, and issue another search if they are not. Described honestly, that is prompting plus looping — no new machinery, no new model capability, just the retriever exposed as something the model can invoke repeatedly. It is powerful, and it is expensive in a way worth naming: every hop replays the entire transcript so far, so a four-hop question pays dive five's prefill bill four times over a context that grows at every step.
And at that point you are no longer building a retrieval pipeline. You are building a loop that decides what to do next, with a tool attached — which is precisely the next dive in this arc.