The Production LLM Stack: Every Component, Mapped to Backend Infrastructure

A demo that calls an LLM API is 30 lines of code. A production system built on that same call grows the same supporting cast your backend services did: a gateway, config management, search infrastructure, job orchestration, tests, observability, and cost controls. None of the components are exotic: every one of them maps to something you already operate. This lesson is the tour, and it doubles as the roadmap for the rest of the track.

The big picture

1. Client / calling serviceyour existing request path
2. ↓
   LLM gateway / router= API gatewayauth, per-team limits, provider failover, spend caps
3. ↓
   Context assembly= config + searchversioned prompts + retrieved RAG chunks + tool schemas
4. ↓
   Model provider: A, B, or self-hosted= upstream servicethe flaky, expensive upstream dependency
5. ↓
   Tool / agent runtime= job orchestrationqueues, bounded retries, idempotency keys, iteration caps
6. ↓
   Evals · tracing · cost controls= CI + APM + FinOpsgolden sets in CI, per-call spans, budgets with attribution

The first five steps are the request path. The last band is the feedback loop that keeps the request path honest. Most teams build the request path first and pay for skipping the feedback loop later.

LLM gateway / routing layer: your API gateway

Every model call goes through one chokepoint instead of every service holding its own provider SDK and API key. The gateway does what Kong or Envoy does for you today: authentication, per-team rate limits, and centralized credentials, plus three LLM-specific jobs: provider failover (Provider A is having an incident; route to B), model routing (send cheap requests to cheap models), and spend caps (a runaway retry loop can't burn $10k overnight). If you'd never let 40 services each hold their own Stripe key, don't let them each hold a model API key. Covered in module 2.

Prompt management: your config management

Prompts are hot-path production config that happens to be written in English. A one-line wording change can shift output quality more than a model upgrade, so prompts get the same treatment as any config that can take down prod: versioned in git, code-reviewed, deployed with an audit trail, and rolled back in one step. Hardcoded prompt strings scattered across services are the new hardcoded connection strings. Covered in module 3.

RAG pipeline: your search infrastructure

Retrieval-augmented generation grounds the model in your data: documents are ingested, chunked, embedded into vectors, stored in a vector index, then retrieved and reranked at query time before being handed to the model as context. If you've run Elasticsearch, this is familiar: ingest/chunk/embed is your ETL and indexing pipeline, retrieval is the query path, reranking is your relevance-tuning layer. The failure modes are familiar too: stale indexes, bad chunking (bad analyzers), and irrelevant top-k results. Retrieval quality, not the model, is usually the bottleneck. Covered in module 4.

Tool execution / agent runtime: your job orchestration

Agents are LLMs that decide which functions to call, in what order, in a loop. Strip the mystique and the operational problem is one you know cold: an agent step is an unreliable worker. It can fail, stall, return garbage, or fire the same side effect twice. So the runtime looks like Sidekiq or Temporal: queues, bounded retries, timeouts per step, idempotency keys on anything with side effects, and a hard cap on loop iterations. The novelty is that your "worker" is non-deterministic; the discipline is the one you already have. Covered in module 5.

Evals: your test suite and CI

Lesson 5 established that exact-match tests die under non-determinism. Evals are what replaces them: golden datasets (curated input/expected-behavior pairs), scored by rules or by LLM-as-judge, run as a regression gate in CI before any prompt or model change ships. The workflow is exactly your test-suite workflow: a prompt PR that drops the eval pass rate from 96% to 84% gets blocked like a PR that breaks the build. Teams without evals ship prompt changes on vibes; it works until it doesn't. Covered in module 6.

Tracing and observability: your APM

Every call gets a trace: full prompt, response, model and version, input/output token counts, TTFT, total latency, and computed cost (the OpenTelemetry span model extended with LLM-specific attributes), and multi-step agent runs nest as parent/child spans exactly like a distributed trace. One new tension: payloads are big (a 100k-token prompt is not a 200-byte span tag), so you need a sampling strategy. Head-sample verbose payload capture at a few percent, tail-sample 100% of errors and outliers, keep lightweight metrics on everything. Covered in module 7.

Cost controls: your FinOps

The per-token cost model from lesson 5 needs the same machinery you point at cloud spend: budgets with alerts, per-feature attribution (which endpoint, team, and customer is driving spend; tags on every call), caching (prompt caching for repeated prefixes, response caching for repeated questions), and model right-sizing (the LLM version of instance right-sizing: most requests don't need the frontier model). Without attribution you'll know the bill doubled but not why. Also module 7.

You don't need all of this on day one

This diagram is where mature systems end up, not where you start. Building all seven components before shipping anything is the same over-engineering as standing up Kubernetes and a service mesh for your first CRUD app.

The minimum viable stack is three pieces:

1. The provider SDK, called directly: no gateway yet. One service, one key, pinned model version.
2. One eval: even 20 golden examples in a script that you run before every prompt change. This is the piece teams skip and regret.
3. Tracing from day one: log prompt, response, tokens, latency, and cost per call. Three extra fields on logging you already do, and it's nearly impossible to retrofit the history you didn't capture.

Everything else is added when a real problem demands it: a second provider forces the gateway, prompt sprawl forces prompt management, "the model doesn't know our data" forces RAG, multi-step workflows force the agent runtime, and the first surprising invoice forces cost controls. The track follows that same order: modules 2 through 7 build the stack the way a real system grows it.

Key takeaways

• A production AI stack is your backend stack with new labels: gateway, config management, search, job orchestration, CI, APM, and FinOps (one LLM-shaped twist each).
• Prompts are hot-path config: version, review, and roll them back like anything else that can take down prod.
• An agent step is an unreliable worker: queues, retries, idempotency keys, and iteration caps apply unchanged.
• Evals are the CI gate for prompt and model changes; shipping without them is shipping without tests.
• Traces need LLM-specific fields (tokens, cost, TTFT) and a payload sampling strategy, because prompts are too big to capture at 100%.
• Start with the minimum viable stack (SDK, one eval, tracing) and add components when a real problem demands them. Modules 2–7 build it in that order.