When the Model Has a Bad Day

Every upstream you've ever depended on fails in two currencies: errors and time. The LLM API adds a third, and it's the one your existing tooling is blind to: success-shaped failures. A 200 OK whose answer was cut off mid-sentence. A 200 OK where the model declined to answer. A 200 OK containing confident, well-formatted garbage.

One thing this lesson is not: a retry tutorial. The mechanics of backoff, which status codes to re-attempt, and how to chain a second model are an operational playbook, and the playbook already exists; read it before or after this lesson. What the playbook takes as given is the question this lesson answers: when the call is not coming back in time, or comes back wrong, what does your API serve instead? That's failure design, it happens at your service boundary, and no amount of retrying substitutes for it.

The failure inventory, from the caller's seat

From your service's point of view, an LLM call ends in one of four buckets:

• Transport and status failures. Connection resets, 429s, 500s, 529s. Loud, familiar, and the only bucket your HTTP client sees; the providers document their taxonomies (Anthropic, OpenAI) and the playbook covers which are worth a re-attempt.
• Hangs. Nothing arrives, or the stream stalls mid-answer. Caught only by the stage-aware timeouts you set: tight on first token, stall guard between deltas, hard deadline overall.
• Success-shaped failures. The 200s that lie, detailed below. Invisible to HTTP-level tooling by definition.
• Slow successes. Technically fine, 25 seconds later. If your user left at second 6, this is a failure that also billed you. The worst bucket, because nothing anywhere logs it as an error.

Two LLM-specific twists make all four buckets more expensive than your instincts expect: failures arrive late (you often burn seconds before learning anything is wrong) and partial failures still cost money (the tokens generated before an error are billed, even when the usage block never arrives).

The 200s that lie

Three shapes, three different correct responses. None of them is "retry."

Truncation. stop_reason: "max_tokens" means the model hit your output cap mid-thought. Lesson 3 taught you to check it like an HTTP status; this is where you decide what the check does. Three defensible moves, in order of preference: serve the partial honestly if the content degrades gracefully (a summary that covers 80% of the thread, marked as such); re-ask with a tighter prompt or bigger budget if correctness demands completeness; or fail the operation if a partial is worse than nothing (extraction jobs, anything parsed downstream). The indefensible move is the default one: passing the truncated text along as if it were whole.

Refusal. The model declines: safety filters, or a misread of an innocent request. Refusals are product events, not errors. Blind re-attempts mostly reproduce the refusal at full price, and hammering a safety refusal is a good way to look like an abuse pattern to the provider. Detect refusals (Anthropic surfaces an explicit refusal stop reason on some models; otherwise it's a classification problem on the output), log them as their own metric, and branch: rephrase-and-retry once for clearly-benign flows, or surface "can't help with that" with an escalation path. A refusal-rate dashboard per feature catches both prompt regressions and emerging abuse.

Well-formed garbage. The response parses, the JSON validates, and the content is wrong: an invented order ID, a policy that doesn't exist. No transport-layer signal exists at all. The honest posture: treat model output as untrusted input, validate what's checkable (module 3 makes this mechanical with schemas), and accept that the remainder is a quality problem owned by evals (module 6), not by this code path.

Error translation: your callers get your contract, not theirs

When the provider fails, the failure you serve must be yours. You already enforce this rule at every other boundary: your API doesn't forward Postgres error strings or Stripe status codes, and it shouldn't forward overloaded_error either. Callers of your API get your status codes, your error shapes, your Retry-After hints, calibrated to what they should do next.

In code, this is one function at the boundary, and it's most of what "handling LLM errors" means in practice:

type LLMFailure =
  | { kind: "overloaded" }       // 429 / 529 / open circuit
  | { kind: "hang"; stage: "ttft" | "stall" | "deadline" }
  | { kind: "truncated"; partial: string }
  | { kind: "refused" }
  | { kind: "invalid_output" }
  | { kind: "provider_error" };  // 4xx config bugs, 5xx internals
 
// One place decides what every LLM failure means for YOUR API's callers.
function toApiResponse(f: LLMFailure, feature: FeatureConfig): ApiResponse {
  switch (f.kind) {
    case "overloaded":
    case "hang":
      return feature.degradable
        ? { status: 200, body: feature.fallbackBody, headers: { "X-Degraded": "true" } }
        : { status: 503, headers: { "Retry-After": "30" } };
    case "truncated":
      return feature.partialsOk
        ? { status: 200, body: { text: f.partial, complete: false } }
        : { status: 502 };
    case "refused":
      return { status: 422, body: { error: "request_not_processable" } };
    case "invalid_output":
    case "provider_error":
      return { status: 502 };    // alarmed on, never retried by clients
  }
}

The point isn't these exact mappings; it's that the mappings are decisions made once, per feature, in review, instead of whatever each call site's catch block improvises at 2am.

The degradation ladder

"Degradable" above is doing heavy lifting, so make it concrete. For every LLM-backed feature, there's a ladder of answers, best to worst, and your job is to know how far down each feature can climb before it should disappear:

Two design rules govern the ladder. First, fail open or fail closed is a per-feature decision: a suggestion widget fails open (hide it, nobody's hurt); an LLM moderation gate fails closed (if the classifier is down, content queues rather than sailing through unmoderated). Classify every feature explicitly. Second, the bottom rung must exist: a page that can't render without the model call has made the model a hard dependency, which is the architecture decision you were warned about, not an incident response.

The kill switch is the ladder's manual override: a feature flag that pins a feature to rung 2 or 3 during an incident, so you stop burning latency budget (and money) on calls you know will fail. Google's SRE book calls this load shedding and graceful degradation; the LLM twist is that shedding also has an immediate invoice benefit, since every shed call is tokens un-bought. During a provider incident, flipping one flag beats watching your p99 melt while a thousand doomed calls time out in sequence. When the incident is instead partial capacity (you're being throttled, not refused), that's a different lever, and the next lesson covers it.

Key takeaways

• LLM calls fail in four buckets: transport/status errors, hangs, success-shaped failures (the 200s that lie), and slow successes. Only the first is visible to HTTP tooling; failures arrive late and cost money even when partial.
• Truncation, refusal, and well-formed garbage each get their own branch: serve honest partials, treat refusals as product events with their own metric, validate output as untrusted input. None of the three is fixed by re-attempting.
• Translate at the boundary: your callers receive your API's statuses and payloads, decided per feature in one reviewed function, never the provider's raw errors.
• Build the degradation ladder per feature: real answer, cached answer, static default, feature hidden. Decide fail-open vs fail-closed explicitly; moderation-style gates fail closed.
• Wire a kill switch that pins a feature down-ladder during incidents; shedding doomed LLM calls saves latency and money at once.
• Exercise degrade paths on a schedule. An untested fallback is a hypothesis, and incidents are a bad time to test hypotheses.