When RAG Is the Right Answer (and When Not)

The production stack lesson listed retrieval as the component that appears the day "the model doesn't know our data" becomes a bug report. Module 3's capstone hit the same wall: a typed LLM function knows only what the prompt carries, so asking it about a customer's subscription tier buys a confident guess.

This module pays that debt. What gets built is a search system: documents in, an index, a query at read time, results assembled into a prompt. Every hard part of it you have already shipped under another name.

Two pipelines you already operate

Retrieval-augmented generation is a write path and a read path, and the model appears only at the end of the second one. The write path reads documents from a source of record, splits them, calls an embedding API, and upserts the vectors into an index. That is an ETL job with a network hop in the middle: batched, idempotent, resumable, monitored on lag and unit cost.

The read path embeds the user's question with the same model, searches the index for the nearest few results, optionally reranks them, assembles the winners into the prompt, and calls the model. That is a query path. Its added p99 belongs to the search hop, its correctness belongs to relevance, and its cost scales with how much text you carry into the prompt.

Two pipelines above and below one shared vector index box of chunks, metadata and embeddings. The top row, labelled write path and your ETL job, runs documents, chunk, embed, upsert, with a line from upsert down into the index. The bottom row, labelled read path and your query path, runs question, embed and search, rerank, prompt and model, with a line from the index into the embed and search stage. — Retrieval is two pipelines sharing one index, and nothing in either row is a component you have not operated before.

The two paths meet at one object, which is where incidents come from: the write path can fall behind reality, and the read path can pick the wrong rows. The 2020 paper that named the technique paired a generator with a dense vector index over Wikipedia; your corpus, unlike a Wikipedia dump, changes. Where the index lives, Postgres or a dedicated service, is a capacity decision covered in the vector store comparison, and it changes neither row.

Four ways to put a fact in front of the model

Retrieval is one of four answers, and the expensive mistake is reaching for it when a WHERE clause would have done. Each branch has a deciding question, and each is about your corpus rather than the model.

A fan from one box reading what is missing? out to four labelled outcomes, each branch carrying its deciding condition: query the database when the question maps to a query, noted SELECT not search; long context when the corpus is small and stable, noted under about 100k tokens; fine-tune when you need different behaviour, noted format and tone not facts; retrieve when the corpus is large or changing, noted needs citations and scoping. A line below the four reads that these combine. — The branch is decided by the shape of the corpus and the shape of the question, not by which technique is newest.

Query the database. Could you write the query by hand? Counts, balances, statuses and dates live in tables with indexes on them, and a sum is not a similarity problem. Costs one round trip you already pay for, and the answer is exact and auditable.
Long context. Does the entire corpus fit in the window, and is it the same corpus for every user? A refund policy, a style guide, a schema description: paste it, version it with the prompt, and skip the index. Costs input tokens on every call, before caching (module 7).
Fine-tune. Is the gap behaviour or facts? Tone, format adherence and a house classification taxonomy respond to training. Facts do not: they change after the run finishes, so a training run per change is stale on arrival.
Retrieve. Is the corpus bigger than a window, changing on someone else's schedule, or scoped per tenant? Add the need to cite a source and retrieval stops being optional. Costs an index to keep fresh plus two extra hops in the request path.

The first branch is the one teams skip, which is how a support bot answers billing questions from prose about billing:

sql

-- "How much did customer 8814 spend last quarter?" is not a retrieval question.
select sum(amount_cents) / 100.0 as spend_usd
from invoices
where customer_id = 8814
  and issued_at >= date_trunc('quarter', now() - interval '3 months')
  and issued_at <  date_trunc('quarter', now());

These combine more often than they compete. A production assistant pastes the policy into the prompt, calls SQL for anything numeric, and retrieves only over the long tail of tickets and runbooks: layers added in cost order, not a bake-off with one winner.

What retrieval does not fix

Retrieval changes what the model knows, not how well it reasons. A model that mangles a multi-step calculation with no context will mangle it with eight chunks of context, and a vague prompt stays vague once you prepend documents. If the failure is bad instructions or a task that needs decomposition, an index is an expensive way to leave it unfixed.

The sharper problem is the wrong document, retrieved confidently. The model cannot tell that chunk three is about a different product line, or that two live versions of the policy are both indexed: it answers from what you handed it in the same fluent register, and the citation you attach makes the wrong answer look sourced. A confidently wrong retrieval is worse than an empty one, so an empty result set has to be a designed branch rather than an accident.

Retrieval quality, not model choice, is usually the bottleneck. Anthropic's contextual retrieval work reports 5.7% of queries failing to surface the right chunk in the top 20 on their internal eval set; prepending generated context before both embedding and lexical indexing cut that to 2.9%, and adding a reranker took it to 1.9%. None of those changes touched the generator. When answers are wrong, check the retrieved set first; the cheapest structural fix is running keyword and vector search together so exact identifiers stop falling through.

The bill for a second system

Retrieval is a second system to operate, and the honest way to price it is in failure modes rather than dollars. The request path gains two dependencies that can time out or degrade, an embedding call and an index query, both in front of a model call you were already waiting on. The deploy story gains an index that must track a database changing without asking you.

Token spend on embeddings is rarely what hurts: OpenAI lists text-embedding-3-small at $0.02 per 1M tokens, a rounding error next to generation. The recurring costs are structural: keeping the index in sync with the source of record, re-embedding the corpus when the model changes, and the input tokens every retrieved chunk adds to every request forever.

One more cost is easy to miss. Retrieved text enters the prompt from a system you do not fully control: untrusted input arriving at the boundary from the other side. If your corpus contains anything a user can write, the context is an injection surface.

Build it in the cheap order

Each layer must earn its place by failing to be enough. Ship the version with no index, measure how often it is wrong, and let the failures name the next layer. That is the shape of this module: embeddings, chunking, ingestion, retrieval quality, then a service behind one typed function.

Answer it without an indexfirstSQL for structured questions, a pasted corpus for small stable ones
Add an index when it stops fittingnextembed, chunk, ingest: the write path as the ETL job it is
Measure the retrieved setbefore tuningscore retrieval before touching the prompt (module 6)
Then pay for qualitylasthybrid search, reranking, contextualized chunks, in that order

Module 5 turns retrieval into a tool the model decides to call, and module 6 makes quality claims testable.

Key takeaways

Retrieval is a write path (chunk, embed, index) plus a read path (embed the query, search, rerank, assemble, call the model): an ETL job and a query path you already operate.
The two paths share one object, the index, so every retrieval incident is either freshness on the write side or relevance on the read side.
Four branches, four deciding questions: could you write the SQL, does the corpus fit the window for every user, is the gap behaviour rather than facts, is the corpus large, changing and per-tenant.
Production systems combine the branches: pasted policy documents, SQL for anything numeric, retrieval for the long tail, added in cost order rather than picked as a winner.
Retrieval does not improve reasoning or repair a vague prompt, and a confidently wrong chunk is worse than an empty result. Empty retrieval must be a designed branch that fails closed.
The real bill is two more request-path dependencies, an index that must track a database changing without you, tokens on every request, and a corpus that becomes an injection surface once users can write into it.

Retrieval Is a Search Problem You Already Own