LLM Fallbacks in Production: Routing, Retries, and Timeouts

An LLM fallback is the model that serves the request when your first choice can't, and the production version needs three things the naive one skips: a routing rule that decides when to switch, an eval that proves the backup is good enough, and a log line recording which model actually answered.

You already know how to wrap a flaky upstream: timeouts, retries with backoff, circuit breaker, fallback. LLM APIs need exactly that playbook with one amendment that changes several defaults: every retry has a price tag. A retried Postgres query costs microseconds of CPU; a retried frontier-model call costs real money and 10+ seconds. Naive retry loops here aren't a reliability bug; they're a money bug that a provider incident can turn into a four-figure invoice overnight.

Here's the playbook, error by error, ending at the fallback chain and the router in front of it. This is what it looks like on the wire when everything that can go wrong does:

The error taxonomy: what to do per status

The two big providers document their error shapes (Anthropic, OpenAI); operationally they collapse into one table:

The bottom two rows are the LLM-specific trap: failures that arrive as 200s. Your HTTP-level retry wrapper will never see them; response-level checks (stop_reason, covered in the API lesson) must. Retrying is the wrong tool for both rows anyway: deciding what your API serves instead is failure design, covered in its own lesson. A third non-retryable family joins the table once outputs are schema-validated: a validation failure gets one repair re-ask with the validator's error in hand, never a transport retry.

Retries: the standard pattern, with a budget

The mechanics are the ones AWS documented years ago (exponential backoff with full jitter), plus two LLM-specific rules:

const RETRYABLE = new Set([429, 500, 529, 503]);
 
async function callWithRetry(req: LLMRequest, maxRetries: number): Promise<LLMResponse> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await callLLM(req, { timeoutMs: req.ttftBudgetMs });
    } catch (err) {
      if (!RETRYABLE.has(err.status) || attempt >= maxRetries) throw err;
      const retryAfterMs = err.retryAfterMs                       // provider said when; believe it
        ?? Math.random() * Math.min(30_000, 1000 * 2 ** attempt); // else: full jitter, capped
      await sleep(retryAfterMs);
    }
  }
}
 
// Sync path (user waiting): callWithRetry(req, 1)
// Async worker:             callWithRetry(req, 3); then the queue's own retry takes over

Rule 1: retries spend real money. Set maxRetries by path: one retry when a user is waiting (past that, your latency budget is gone anyway; degrade instead), two or three in queue workers where the queue's dead-letter machinery is the final backstop. Meter retry spend: a retries × cost metric per feature catches the worker that's quietly re-buying the same generation all night.

Rule 2: retry the call, not the side effects. LLM calls aren't idempotent in output (same input, different words; that's the sampling lesson) but they're side-effect-free, so re-calling is safe. The danger is downstream: if the model's output triggers an email, a refund, a tool execution, idempotency-key those operations. Generation is retryable; actions are not.

A 429 deserves one more note: it means back off, not try harder. Under a provider incident, every client's naive retry loop synchronizes into a thundering herd: the cascading-failure pattern with a billing meter attached. retry-after is the provider telling you the herd schedule; honor it. Better still, throttle yourself below the published limits so most 429s never fire at all.

Timeouts: two stages, not one

A flat timeout is wrong twice for LLM calls, because latency has two phases: time-to-first-token (queueing + prompt processing, usually sub-second, occasionally stuck) and generation (legitimately tens of seconds for long outputs).

• TTFT timeout, tight (~10s): nothing arrived → the call is probably wedged. Abort, retry once, or fall back. This is your real health signal.
• Total-duration timeout, generous (60–120s): protects against runaway streams. If you hit it regularly, the fix is a smaller max_tokens or a terser prompt, not a bigger timeout.
• Inter-token stall timeout (streaming, optional): no delta for ~15s mid-stream → the stream died; end it cleanly rather than leaving the user watching a frozen cursor.

Set these from your caller's budget, not the provider SDK defaults, which are effectively "forever."

Fallbacks: the chain, and its two flavors

When retries exhaust, the next move is a different model. Two flavors, different purposes:

Cross-provider, same tier: for availability. Provider A is down; a comparable model from provider B serves. Costs you a second integration and prompt portability (keep prompts provider-neutral or maintain two variants; the two-dialect reality makes this cheaper than it sounds).

Same provider, smaller model: for overload and cost pressure. Opus-class is 529ing or your spend cap tripped; Sonnet- or Haiku-class serves a good-enough answer. This is model routing wearing its incident-response hat.

const CHAIN = [
  { provider: "anthropic", model: "claude-sonnet-5" },
  { provider: "openai",    model: "gpt-5" },            // availability fallback
  { provider: "anthropic", model: "claude-haiku-4-5" }, // degraded-quality fallback
];

Three rules that keep the chain honest. Eval the fallback before you need it: a fallback that's never been run against your eval set isn't a fallback, it's a hope. Log which link served every request: quality complaints during an incident are unexplainable without it. Put the chain in one place: the gateway chokepoint, not copy-pasted per call site.

And below the whole chain sits the real floor: degrade the feature. Suggestion returns null, summary shows "unavailable," the page still renders. An LLM feature that can't degrade isn't a feature, it's a dependency.

Fallback routing: pick the model before the failure, not after

A chain is reactive and fixed: try A, then B, then C, in that order, every time. Fallback routing puts a decision in front of the chain, per request, using signals you are already collecting. The chain answers "what next"; the router answers "what first, and why."

Four signals are worth routing on, and the order matters:

// Capability is a gate, not a preference: falling back to a model that can't
// fit the context isn't a fallback, it's a second failure with extra latency.
function route(req: LLMRequest, health: HealthByProvider): ModelRef[] {
  const capable = CHAIN.filter((m) => supports(m, req));
  const healthy = capable.filter((m) => !health[m.provider].circuitOpen);
  const pool = healthy.length > 0 ? healthy : capable;   // all sick: try anyway, degraded
 
  return req.budgetRemainingUsd < req.estimatedCostUsd
    ? pool.filter(isCheapTier)                            // downshift, don't fail
    : pool;
}

Two failure modes specific to routing. Sticky within a conversation: if turn 4 gets answered by a smaller model than turns 1-3, the voice and format shift mid-thread and users read it as a bug. Pin the chosen model for the life of a session unless it goes unhealthy. Never route on latency alone: the fastest model is usually the weakest, so a p95 spike will quietly migrate all your traffic to your worst model and the graphs will look great.

Circuit breaker: stop paying to fail

Retries and fallbacks handle one request; a circuit breaker handles the incident. Track error rate per provider; past a threshold (say, 50% over 30s), open the circuit (route straight to fallback or degrade, skip the doomed-and-billed attempts) and let periodic probes close it when the provider recovers. Any standard breaker library works unchanged; the LLM twist is that an open circuit saves money, not just latency, which makes the business case for wiring it unusually easy.

If you remember one thing: classify before you retry (most 4xx-class failures can only be fixed, not retried, and some failures arrive as 200s), spend retries like the money they are, and make the last link in every chain a graceful degrade. The track builds the gateway that gives all of this one home.

Sources & further reading: Anthropic API errors · OpenAI error codes · Exponential backoff and jitter (AWS) · Google SRE: cascading failures