Fixing Retrieval, Not the Model

A support assistant answers a refund question with a confident, wrong policy. The prompt is fine, the output validated against its schema, the model the newest one the provider sells. Neither a newer model nor more instructions will fix it.

Retrieval is a search problem wearing a machine learning costume. When a RAG answer is wrong, the defect is almost always in the ranked list you handed the model. You already debug this: a query returns the wrong rows, so you read the plan.

Four failures that all look like one bad answer

Every bad retrieval decomposes into one of four failures, and no fix works on more than one of them.

• Not indexed. The crawler skipped the page, the PDF parsed to whitespace, or the row changed after the last sync. Ranking cannot retrieve a chunk that does not exist.
• Wrong chunk. The right document is indexed, but the passage that matched is the adjacent paragraph or a heading with no body under it. A chunking defect: answers come back topically correct and factually empty.
• Buried. The right chunk is in the candidate set at rank 14 and you cut at 5. Nothing is missing or malformed; the ordering is wrong.
• Flooded. You passed 20 passages, the answer sat in one, and the other 19 diluted it. You paid input tokens and prefill latency for the dilution.

"The model hallucinated" is not a diagnosis anyone can act on. It is usually failure one or three: a fluent text predictor filling a gap in the context it was handed.

Diagnose before you tune

The diagnosis costs two queries, and it tells you which lesson you are actually in. Take one question you know is answered wrong and one passage that should answer it.

Look the passage up by exact text, not by vector search. If it is not there you have an ingestion bug. If it is there, run the real query with k = 100 and find its rank.

Log the retrieved chunk ids and scores on every request, the way you keep a slow query log.

The fix ladder, cheapest first

Work the rungs in cost order, because the cheap ones fix the common failures. The first rung is hybrid search, lexical and vector search fused. Pure vector search is weakest on what users actually type, an error code, a SKU, a version string, where only a literal token match will do. That guide owns the SQL and the fusion.

The second rung is reranking, the direct fix for buried. Retrieval compares two vectors computed independently, which is why the corpus side can be precomputed, and the index over them is approximate. A cross-encoder reads query and passage together in one forward pass: far more accurate, far too slow over a whole corpus. Retrieve wide, rerank narrow: 100 to 200 candidates in, 5 out.

Neither OpenAI nor Anthropic ships a standalone reranker, so this rung is a vendor call. Cohere's POST https://api.cohere.com/v2/rerank takes rerank-v4.0-pro or rerank-v4.0-fast (32k context, relevance scores normalized to 0 through 1); its best practices put the hard ceiling at 10,000 documents and its API reference recommends under 1,000. Voyage's POST https://api.voyageai.com/v1/rerank takes rerank-2.5 or rerank-2.5-lite, and its API reference caps a call at 1,000 documents, an 8,000-token query, and 32,000 tokens for the query plus any single document. AWS Bedrock and Google Cloud expose one natively.

const res = await fetch("https://api.cohere.com/v2/rerank", {
  method: "POST",
  headers: { authorization: `Bearer ${process.env.COHERE_API_KEY}`,
             "content-type": "application/json" },
  body: JSON.stringify({
    model: "rerank-v4.0-fast",
    query,                                 // truncated above 16,384 tokens
    documents: candidates.map((c) => c.content),  // keep this under 1,000
    top_n: 5,
  }),
});
const { results } = await res.json();       // [{ index, relevance_score }]
const top = results.map((r) => candidates[r.index]);  // ids survive the hop

Neither Cohere nor Voyage publishes an absolute latency figure, only relative claims, so the number in your budget has to be one you measured on a serial hop before the first token. Rungs three and four, metadata filters and top-k, are free, but they fix scope leakage and flooding, not burial.

Filters belong inside the query, not after it

Permission-scoped retrieval is a security boundary, not a search feature, and the scope has to be applied inside the search. Filter after retrieval and those rows have already entered your process, your traces and your logs, and every new call site is a fresh chance to forget. Tenant id and access scope are a WHERE clause like any other, sourced from the authenticated session, never from the request body, the query text, or the model.

pgvector applies your WHERE clause after the approximate index scan, so per its README, a condition matching 10% of rows against the default hnsw.ef_search of 40 leaves about 4 rows instead of 50, which reads like a ranking problem. The documented fixes are iterative index scans (pgvector 0.8.0 and later), a partial index per scope, or a bigger candidate list.

BEGIN;
SET LOCAL hnsw.ef_search = 200;              -- default is 40
SET LOCAL hnsw.iterative_scan = strict_order; -- pgvector 0.8.0+
SELECT id, document_id, content
FROM doc_chunks
WHERE tenant_id = $1 AND acl_group = ANY($2) -- scope, from the session
ORDER BY embedding <#> $3                     -- negative inner product: ASC is nearest
LIMIT 50;
COMMIT;

Context assembly is a budget, not a bucket

Top-k is not a retrieval parameter, it is a line item against the token budget. Twenty chunks at 400 tokens each is 8,000 input tokens on every request, with prefill latency attached. That is the price of "just retrieve more."

More context also costs accuracy: the correct passage competes for attention with every near-duplicate beside it. So deduplicate before assembly, overlapping windows from the same document, repeated page headers and footers. Order deterministically, or no before-and-after comparison means anything.

Then carry the source ids through the whole path, from search result into the prompt and back out with the answer. That is what lets the answer cite, and what turns "the bot said something wrong" into a chunk id you can look up.

The number that tells you it worked

Every fix above is a hypothesis until recall@k on a golden set moves. Label real questions with the chunk id that answers each one, then measure the share whose answering chunk lands in the top k. Run it before and after each rung.

Recall@k measures the ranked list, not answer quality, so it moves without asking a model to grade prose. The golden set, the metric family, and the CI gate that blocks a regression belong to the evaluation guide; answer-level evaluation and tracing arrive in module 6.

Key takeaways

• Bad retrieval is one of four failures: not indexed, wrong chunk, buried below the cut, flooded with context. Each has its own fix, and "the model hallucinated" is usually the first or the third.
• Diagnose with two queries: an exact-text lookup to prove the passage is indexed, then its rank in the top 100 to split a chunking problem from a ranking problem.
• Fix in cost order. Hybrid search is the first rung, because pure vector search is weakest on error codes, SKUs and version strings, the literal tokens users type.
• Reranking is the fix for buried: retrieve 100 to 200 candidates, score query and passage together, keep 5. Keep the call under 1,000 documents, and measure the extra serial hop yourself, because no vendor publishes a latency figure.
• Top-k is a token line item: twenty chunks at 400 tokens is 8,000 input tokens per request plus prefill. Deduplicate, order deterministically, and carry source ids through for citation and lookup.
• Permission filters run inside the search, sourced from the session, and the cache key carries scope too. pgvector applies the filter after the approximate scan, so a 10% selective condition against the default hnsw.ef_search of 40 leaves about 4 rows.