LLM API Retries, Timeouts, and Fallbacks: A Production Playbook

You already know how to wrap a flaky upstream: timeouts, retries with backoff, circuit breaker, fallback. LLM APIs need exactly that playbook with one amendment that changes several defaults: every retry has a price tag. A retried Postgres query costs microseconds of CPU; a retried frontier-model call costs real money and 10+ seconds. Naive retry loops here aren't a reliability bug; they're a money bug that a provider incident can turn into a four-figure invoice overnight.

Here's the playbook, error by error.

The error taxonomy: what to do per status

The two big providers document their error shapes (Anthropic, OpenAI); operationally they collapse into one table:

The bottom two rows are the LLM-specific trap: failures that arrive as 200s. Your HTTP-level retry wrapper will never see them; response-level checks (stop_reason, covered in the API lesson) must. Retrying is the wrong tool for both rows anyway: deciding what your API serves instead is failure design, covered in its own lesson.

Retries: the standard pattern, with a budget

The mechanics are the ones AWS documented years ago (exponential backoff with full jitter), plus two LLM-specific rules:

const RETRYABLE = new Set([429, 500, 529, 503]);
 
async function callWithRetry(req: LLMRequest, maxRetries: number): Promise<LLMResponse> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await callLLM(req, { timeoutMs: req.ttftBudgetMs });
    } catch (err) {
      if (!RETRYABLE.has(err.status) || attempt >= maxRetries) throw err;
      const retryAfterMs = err.retryAfterMs                       // provider said when; believe it
        ?? Math.random() * Math.min(30_000, 1000 * 2 ** attempt); // else: full jitter, capped
      await sleep(retryAfterMs);
    }
  }
}
 
// Sync path (user waiting): callWithRetry(req, 1)
// Async worker:             callWithRetry(req, 3); then the queue's own retry takes over

Rule 1: retries spend real money. Set maxRetries by path: one retry when a user is waiting (past that, your latency budget is gone anyway; degrade instead), two or three in queue workers where the queue's dead-letter machinery is the final backstop. Meter retry spend: a retries × cost metric per feature catches the worker that's quietly re-buying the same generation all night.

Rule 2: retry the call, not the side effects. LLM calls aren't idempotent in output (same input, different words; that's the sampling lesson) but they're side-effect-free, so re-calling is safe. The danger is downstream: if the model's output triggers an email, a refund, a tool execution, idempotency-key those operations. Generation is retryable; actions are not.

A 429 deserves one more note: it means back off, not try harder. Under a provider incident, every client's naive retry loop synchronizes into a thundering herd: the cascading-failure pattern with a billing meter attached. retry-after is the provider telling you the herd schedule; honor it. Better still, throttle yourself below the published limits so most 429s never fire at all.

Timeouts: two stages, not one

A flat timeout is wrong twice for LLM calls, because latency has two phases: time-to-first-token (queueing + prompt processing, usually sub-second, occasionally stuck) and generation (legitimately tens of seconds for long outputs).

• TTFT timeout, tight (~10s): nothing arrived → the call is probably wedged. Abort, retry once, or fall back. This is your real health signal.
• Total-duration timeout, generous (60–120s): protects against runaway streams. If you hit it regularly, the fix is a smaller max_tokens or a terser prompt, not a bigger timeout.
• Inter-token stall timeout (streaming, optional): no delta for ~15s mid-stream → the stream died; end it cleanly rather than leaving the user watching a frozen cursor.

Set these from your caller's budget, not the provider SDK defaults, which are effectively "forever."

Fallbacks: the chain, and its two flavors

When retries exhaust, the next move is a different model. Two flavors, different purposes:

Cross-provider, same tier: for availability. Provider A is down; a comparable model from provider B serves. Costs you a second integration and prompt portability (keep prompts provider-neutral or maintain two variants; the two-dialect reality makes this cheaper than it sounds).

Same provider, smaller model: for overload and cost pressure. Opus-class is 529ing or your spend cap tripped; Sonnet- or Haiku-class serves a good-enough answer. This is model routing wearing its incident-response hat.

const CHAIN = [
  { provider: "anthropic", model: "claude-sonnet-4-5" },
  { provider: "openai",    model: "gpt-5" },            // availability fallback
  { provider: "anthropic", model: "claude-haiku-4-5" }, // degraded-quality fallback
];

Three rules that keep the chain honest. Eval the fallback before you need it: a fallback that's never been run against your eval set isn't a fallback, it's a hope. Log which link served every request: quality complaints during an incident are unexplainable without it. Put the chain in one place: the gateway chokepoint, not copy-pasted per call site.

And below the whole chain sits the real floor: degrade the feature. Suggestion returns null, summary shows "unavailable," the page still renders. An LLM feature that can't degrade isn't a feature, it's a dependency.

Circuit breaker: stop paying to fail

Retries and fallbacks handle one request; a circuit breaker handles the incident. Track error rate per provider; past a threshold (say, 50% over 30s), open the circuit (route straight to fallback or degrade, skip the doomed-and-billed attempts) and let periodic probes close it when the provider recovers. Any standard breaker library works unchanged; the LLM twist is that an open circuit saves money, not just latency, which makes the business case for wiring it unusually easy.

If you remember one thing: classify before you retry (most 4xx-class failures can only be fixed, not retried, and some failures arrive as 200s), spend retries like the money they are, and make the last link in every chain a graceful degrade. The track builds the gateway that gives all of this one home.

Sources & further reading: Anthropic API errors · OpenAI error codes · Exponential backoff and jitter (AWS) · Google SRE: cascading failures