How to Evaluate a RAG Pipeline: Metrics That Can Gate a Deploy

Score a RAG pipeline in two halves: retrieval metrics (recall@k, precision@k, MRR, nDCG) against labelled relevant documents, and generation metrics (faithfulness, answer relevance) against whatever the retriever actually returned. A blended end-to-end number is the evaluation mistake that hides the most, because it lets a large retrieval regression through as a small blended one. This guide is the two gates, the golden set they read, and the script that exits non-zero.

Why an end-to-end score hides regressions

A capable generator is a very good liar about retrieval quality. A chunking change ships on a Tuesday. Recall@5 falls, say from 0.82 to 0.70, twelve points of evidence that stopped reaching the context window. The end-to-end answer score moves from 0.79 to 0.77.

The gate allows a 3 point drop, so the build stays green and the change ships. Users find it two weeks later, and the report is never "recall dropped", it is "answers got vaguer".

The masking is arithmetic. A blended score averages both halves, and a strong model covers for thin context by paraphrasing what it did receive. The stronger your generator, the more retrieval damage it absorbs before the blended number notices.

Split the pipeline in two

The two halves cannot share a metric because they cannot share an input. Retrieval is scorable only against ground truth labelled in advance: for this query, these chunk ids are relevant. Generation is scorable only against the context actually retrieved on this run.

Four retrieval metrics cover nearly every case, three of them formalized in chapter 8.4 of Manning, Raghavan and Schütze:

• recall@k: of the documents labelled relevant, the fraction appearing in the top k. This is the metric to gate on, because evidence that never arrives is unusable by any prompt or model.
• precision@k: of the top k returned, the fraction that are relevant.
• MRR: the reciprocal rank of the first correct result, zero when there is none, averaged over queries. From the TREC-8 question answering track (Voorhees, 1999).
• nDCG@k: discounted cumulative gain, normalized so a perfect ranking scores 1. The only one of the four handling graded relevance (Järvelin and Kekäläinen, 2002).

Generation metrics ask a different question: given this context, is the answer grounded and on topic? Ragas supplies the vocabulary, and its class names are worth getting right:

• Faithfulness splits the answer into claims and checks each against the retrieved context, scoring supported over total. FaithfulnesswithHHEM swaps the LLM verifier for Vectara's HHEM-2.1-Open classifier, cheaper to run continuously.
• AnswerRelevancy (legacy name ResponseRelevancy) reverse-engineers questions from the answer and averages their cosine similarity to the original: intent match, not truth. ContextPrecision is rank-weighted, so an irrelevant chunk at position 1 costs more than one at position 10, and LLMContextRecall needs a gold answer rather than gold documents. Both context metrics also ship non-LLM variants (IDBasedContextPrecision, NonLLMContextRecall) that match ids or strings, so labelled chunk ids buy you the metric without the judge.

When recall@k is the number that moved, the fix lives in the retrieval half: hybrid search first, then the rest of the retrieval-quality ladder.

Building the golden set

Fifty to two hundred real queries beat ten thousand synthetic ones, and effort is not the reason. Questions generated from your own chunks are written backwards from the answer, so the answering chunk is trivially retrievable and the set inherits the retriever's blind spots.

Harvest from production logs instead, sampling the shapes that break things: identifier lookups, multi-hop questions needing two documents, near-duplicates where the wrong one is plausible, and questions the corpus cannot answer, where correct behaviour is abstention. Keep the hard cases and discard the average ones, because a set of average queries reports the average, and the average never visibly regresses.

Label once, by hand: a human marks which chunk ids answer each query, roughly a day per hundred queries. Version it in git as a fixture reviewed in pull requests, so a threshold change and a label change both appear in a diff.

export type GoldenCase = {
  id: string;                    // stable, quotable in a review comment
  query: string;                 // verbatim from a production log
  relevantChunkIds: string[];    // labelled by a human, best first
  referenceAnswer: string;       // gold answer, for context recall and grading
  shape: "identifier" | "paraphrase" | "multi-hop" | "unanswerable";
  labelledAt: string;            // ISO date, so stale labels are visible
};

LLM-as-judge without fooling yourself

A judge model is a versioned dependency, not an instrument. Pin the model id and the judge prompt together, because a silent judge upgrade moves every score at once and reads exactly like a product regression. Model versions carry deprecation clocks, so score the golden set with both judges before cutting a migration over.

Calibrate before trusting a threshold. Hand-label 100 to 200 examples, measure how often the judge agrees, and let that rate set the gate's slack. The quoted "over 80% agreement with humans" figure is real but narrow: Zheng et al. measured it for a strong judge on MT-Bench and Chatbot Arena, against human-to-human agreement on the same data.

That paper also names the failure modes: position, verbosity and self-enhancement bias. Separate work supplies the mitigations: swap the ordering and average on any pairwise comparison, and never let the judge be the model under test, because models that recognize their own output rate it higher. Prefer narrow rubrics. "Is each claim supported by the context" is checkable claim by claim; "rate this answer 1 to 10" is a vibe with a decimal point.

The CI gate, with code

A check that cannot fail the build is a dashboard with extra steps. Every pull request gets a fast subset, roughly 50 queries, retrieval only, inside the couple of minutes a PR check can afford. Release runs the full 200, generation and judge included.

Set thresholds from the current baseline, not from ambition. Check in a baseline.json written by the last green release, gate on the delta, and take the tolerance from measured spread: rerun three times on unchanged code. Absolute targets ("recall@5 above 0.90") either never pass or never fail.

// eval/gate-retrieval.ts: CI entry point. A non-zero exit fails the build.
import { readFileSync } from "node:fs";
const read = (p: string) => JSON.parse(readFileSync(p, "utf8"));
const TOLERANCE = 0.02;                        // from measured run-to-run spread
const baseline = read("eval/baseline.json");   // written by the last green release
const cases: GoldenCase[] = read("eval/golden.fast.json");
const scored = cases.filter((c) => c.relevantChunkIds.length > 0);
 
async function main() {
  let sum = 0;
  for (const c of scored) {                    // unanswerable cases have no recall
    const top = await retrieve(c.query, 5);    // your retriever, untouched
    const ids = new Set(top.map((d) => d.id));
    const found = c.relevantChunkIds.filter((id) => ids.has(id)).length;
    sum += found / c.relevantChunkIds.length;  // per-query recall@5
  }
  const recall = sum / scored.length;
  const drop = baseline.recallAt5 - recall;
  if (!Number.isFinite(recall) || drop > TOLERANCE) {
    console.error(`recall@5 ${recall.toFixed(3)} vs ${baseline.recallAt5}: -${drop.toFixed(3)}`);
    process.exit(1);                           // the gate, not a warning
  }
}
main();

Dividing by scored.length matters: an unanswerable case has no relevant ids, so leaving it in makes recall NaN, and NaN > TOLERANCE is false, a gate that can never fail. Those cases get their own assertion instead, that the pipeline abstains.

The generation gate is a second, independent assertion on mean faithfulness over the same subset, with its own tolerance and message. Gate on that mean and never on a single verdict, pin the judge's temperature and seed, and measure the judge's own spread over three runs on unchanged code, so the tolerance absorbs judge variance and not just retriever variance. Prompt A/B testing and rollback is the deploy-side sibling.

After the gate: production traces as trends

Live traffic has no labels, so it gets trends and never a pass/fail. A trace does hold the retrieved context and the answer, so faithfulness and answer relevance can be scored on one or two percent of traffic.

Emit those scores as telemetry rather than into a private eval database. OpenTelemetry's GenAI conventions define a gen_ai.evaluation.result event carrying gen_ai.evaluation.name, gen_ai.evaluation.score.value, a low-cardinality gen_ai.evaluation.score.label and a free-form gen_ai.evaluation.explanation. Retrieval spans have their own type with gen_ai.retrieval.top_k and gen_ai.data_source.id, named by data source rather than by model.

Two caveats: every GenAI convention there is still marked Development in a repo with no tagged release, and several widely copied attributes are already gone (gen_ai.system became gen_ai.provider.name, and prompt and completion token counts became gen_ai.usage.input_tokens and gen_ai.usage.output_tokens).

Alert on week-over-week shape rather than a fixed floor, and promote the low scorers into the golden set.

The decision table

The shape that survives contact with a real team is small: a hand-labelled set of real queries versioned in git, a pinned judge, two assertions that can each fail a build on their own, and thresholds carried in a baseline file rather than in someone's ambition. Sampled traces then feed the next labelled case, which is how a regression that shipped once stops being able to ship twice.

Sources & further reading: Introduction to Information Retrieval, ch. 8.4 · Ragas metrics reference · Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023) · OpenTelemetry GenAI events