Capstone: A RAG Service Behind One Function

Module 3 ended with a function: a versioned prompt in, a validated type out, transport and parsing and repair handled once behind a signature. This module added an index, a chunking scheme, and an ingestion pipeline. None of it changes the shape of what the rest of your codebase imports.

The capstone is one export, answerWithContext(question, scope). A retrieval-backed answer has a failure mode a completion does not: answering anyway when the corpus had nothing to say. The original RAG paper bolted a non-parametric index onto a parametric model. The engineering problem is that when the index comes back empty, the parametric half is still willing to talk.

One function, one return type

The signature is the design document, and the return type is where the moral lives. A RAG call has two outcomes worth returning, a grounded answer and a refusal; transport and model errors throw, and the retry layer absorbs them. If the function returns string, callers render refusals as answers, because a refusal in prose is indistinguishable from a short reply.

export type Citation = { docId: string; chunkId: string; score: number };  // ids die on a re-chunk
export type RefusalReason = "no_candidates" | "below_threshold" | "ungrounded_answer";
export type Scope = { tenantId: string; corpus: string };
 
export type RagResult =
  | { status: "answered"; answer: string; citations: Citation[] }
  | { status: "refused"; reason: RefusalReason; detail: string };
 
// Callers import this. They never see an embedding, an index, or a prompt.
export async function answerWithContext(
  question: string,
  scope: Scope,
): Promise<RagResult> {
  // ...retrieve, gate, generate, ground-check. Body below.
}

The union is discriminated, so TypeScript forces every caller to handle refused before touching answer. Citations are structured rows, not a footnote string, because the UI links them and the grounding check verifies them. And scope carries tenantId as a required parameter, so the tenant filter cannot be forgotten at a call site.

The request path, end to end

Four hops sit between the question and the answer, and each is a place the request can end early. Embed the question, search the index, rerank the candidates, then apply a score threshold. Only past that gate do you assemble a prompt and call the model.

Two hops are third-party API calls (embedding, then rerank) before the expensive model call starts, so the p99 of answerWithContext is not the p99 of the completion. The strategy field earns its place too: the first fix for bad recall is usually hybrid search over the same Postgres table, so the trace has to name the strategy that served the failed request.

Fail closed, or do not ship it

When nothing clears the threshold, the function returns a refusal and no prompt is ever built. The tempting version sends the question with an empty context block and "use the context if it helps," which converts a retrieval miss into a fluent answer from training data. You end up with a system whose worst answers are its most confident, and whose failures look like successes in the logs.

const MIN_SCORE = 0.35;   // reranker relevance, higher is better; retune per scorer
const span = trace.getActiveSpan()!;
const candidates = await search(question, scope, { k: 200, strategy: "hybrid" });
const ranked = await rerank(question, candidates, { topN: 5 });
const kept = ranked.filter((c) => c.score >= MIN_SCORE);
 
span.setAttributes({
  "gen_ai.operation.name": "retrieval",
  "gen_ai.data_source.id": scope.corpus,
  "gen_ai.retrieval.top_k": 200,
  "rag.candidate_count": candidates.length,
  "rag.kept_count": kept.length,
  "rag.refused": kept.length === 0,
});
 
if (candidates.length === 0) {
  return { status: "refused", reason: "no_candidates",
    detail: "index returned nothing in scope" };
}
if (kept.length === 0) {
  return { status: "refused", reason: "below_threshold",
    detail: `${candidates.length} candidates, top score ${ranked[0].score}` };
}
return await generateGrounded(question, kept, span);  // the prompt sees only kept

This is module 3's untrusted-input posture read from the other direction: a corpus miss is a data problem, and the model is the last component you want papering over data problems.

The grounding check catches structure, not truth

After generation, verify that every claim in the answer points at a passage you actually retrieved, and refuse if it does not. Ask for structured output rather than prose: claim objects, each carrying its text and the chunkId it came from, using module 3's schema mechanics. The check is then set membership: every cited id must appear in kept, and every sentence must carry a citation. A claim citing a chunk outside kept is fabricated, and returns ungrounded_answer; answer is assembled from the claims that pass.

The check is cheap and deterministic, which is why it is limited.

Cite-or-refuse proves the answer is anchored, not that the anchor holds. Whether a passage entails the claim it was cited for is a judgment call, scored offline as faithfulness against a judge model and a labelled set: module 6 and the RAG evaluation metrics guide, not the request path.

The trace fields that earn their place at 3am

An answer you cannot reconstruct is a bug report you cannot close, because "the bot said something wrong yesterday" is actionable only if you can replay the ranked set the model saw. OpenTelemetry's GenAI conventions define retrieval and embeddings spans with names for most of this. Every gen_ai.* attribute there is still marked Development, so expect renames; borrowing a convention still beats inventing one.

• gen_ai.retrieval.query.text and the tenant id: the exact question, not a paraphrase from a support ticket. It is Opt-In because questions carry personal data, so gate it like request-body logging.
• gen_ai.data_source.id, which also names the span (retrieval spans are {operation} {data_source.id}; inference and embeddings spans use the model). It answers "did we search the right corpus," a depressingly common root cause.
• gen_ai.retrieval.top_k next to the candidate count. Asking for 40 and getting 4 back signals a filter applied after an approximate index scan, and neither number shows it alone.
• gen_ai.retrieval.documents, the ids and scores. Opt-In, and the highest-value field here: the ranked set separates a retrieval failure from a generation failure in seconds.
• Rerank duration on its own span. Cohere and Voyage publish no absolute latency figures for their rerankers, so your histogram is the only number that exists.
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (prompt_tokens and completion_tokens are deprecated spellings), plus gen_ai.request.model, and gen_ai.embeddings.dimension.count on the embeddings span. Change the embedding model and the index silently means something else, so its version belongs beside the answer.
• Cost and the refusal flag, as your own attributes, since neither has a sanctioned name. Derive cost per span from tokens and your rate card, and keep rag.refused with its reason: refusal rate is the service's health metric.

Wire one more convention early. A gen_ai.evaluation.result event carries an eval name, a score, a label, and an explanation: the sanctioned place to hang a judge's verdict on the trace that produced it.

Where this goes next

Two things happen to this function later. Its signature makes it a tool an agent can call as one step among several (module 5), and its threshold becomes a gate when the golden set that tuned MIN_SCORE becomes a CI check that fails the build on a recall regression (module 6 and the eval guide).

Retrieval is a query with a latency budget, a tenant filter, and a failure mode, wrapped in a signature so the rest of the system never learns it is there.

Key takeaways

• A RAG call returns an answer with citations or a refusal with a reason, so make it a discriminated union. A bare string makes callers render refusals as answers.
• Put the tenant in the signature. scope as a required parameter means the compiler enforces the multi-tenant filter a code review would only hope for.
• Fail closed: when nothing clears the threshold, return the reason and build no prompt at all. An empty context block plus permission to answer turns a retrieval miss into a confident hallucination.
• The threshold is config, not a literal: raising it trades refusals for wrong answers, so dashboard refusal rate next to error rate.
• Cite-or-refuse is a set-membership check: every claim cites a chunk that was in the context. It proves the answer is anchored, not that the anchor supports it.
• Trace the ranked doc ids and scores, k versus candidate count, both token counts, and both model versions. Without the ranked set you cannot separate a retrieval failure from a generation failure.