Rate Limits Are a Capacity Contract

You'd never size a service to open unlimited Postgres connections and then treat "too many clients" errors as bad luck. You provision a pool, you know its size, and the pool makes callers wait instead of stampeding the database. Provider rate limits deserve the same posture: they are your provisioned capacity, published per model, and a 429 just means you spent capacity you never planned.

This lesson is the planning: what the meters actually measure, how to see them before they bite, how to make your own services queue instead of erroring, and when the answer is "buy a bigger pool." (What to do with a 429 after it happens, backoff and all, is the playbook guide's department; the goal here is needing that page rarely.)

Two meters, enforced separately

Every mainstream provider meters you at least two ways at once (OpenAI, Anthropic): requests per minute and tokens per minute, and exceeding either one 429s you. Daily variants and per-tier quotas stack on top; some providers meter input and output tokens separately. The two meters trip on opposite workload shapes:

Tokens are the meter that matters for most production workloads, because tokens are the real unit of everything in these systems: cost, latency, and now capacity too.

Limits come in tiers: as of mid-2026, both major providers scale your RPM/TPM automatically with spend history and account age, with published tier tables, and let you request raises beyond that. The practical consequence: your limits are a function of your history, so a brand-new account cannot launch a high-traffic feature tomorrow. Capacity, like credit, gets built.

The headers: observability before the first 429

Both providers tell you where the meters stand on every successful response. OpenAI sends x-ratelimit-limit-requests, x-ratelimit-remaining-requests, x-ratelimit-remaining-tokens and friends; Anthropic sends anthropic-ratelimit-requests-remaining, anthropic-ratelimit-tokens-remaining, each with limits and reset timestamps. Almost nobody reads them, which is like ignoring pool utilization metrics until "too many clients" pages you.

Reading them is three lines inside the client you already have, feeding the log line you already write:

function captureRateLimit(res: Response) {
  const h = res.headers;
  metrics.gauge("llm.ratelimit.tokens_remaining", {
    value: Number(h.get("anthropic-ratelimit-tokens-remaining") ?? NaN),
    limit: Number(h.get("anthropic-ratelimit-tokens-limit") ?? NaN),
  });
}

Graph tokens_remaining / tokens_limit and alert when the floor of that ratio dips under ~20% during peak. Now "we're going to hit the limit next month at this growth rate" is a dashboard read, and the tier-raise request goes in weeks before launch day instead of during the incident.

Throttle yourself, or the provider does it for you

The core move, same as a connection pool: put a limiter in front of the provider call, sized from your published limits, so excess demand waits in your process instead of turning into 429s, retries, and the thundering-herd bill. Waiting 400ms is invisible to a user; an error-retry cycle is not, and it burns quota to accomplish nothing.

A minimal token-bucket that understands both meters:

// Sized from the provider dashboard. Leave ~20% headroom for retries and drift.
const bucket = new TokenBucket({
  tokensPerMinute: 320_000,     // 80% of a 400k TPM limit
  requestsPerMinute: 1_600,     // 80% of 2k RPM
});
 
export async function callLLM(req: LLMRequest): Promise<LLMResponse> {
  const estimate = req.inputTokens + req.maxTokens;  // mirror provider accounting
  await bucket.take({ tokens: estimate, requests: 1 }); // waits, never throws
  try {
    return await provider.complete(req);
  } finally {
    bucket.reconcile(req, actualUsage(req));  // return the unused max_tokens reserve
  }
}

The two subtleties are in the comments: admit on the estimate (input plus max_tokens, mirroring the provider's own accounting) and reconcile afterward with actual usage so unused reserve returns to the bucket. Libraries like bottleneck or p-limit cover the plumbing; the sizing logic is yours. For traffic that can tolerate minutes of delay, the stronger version of "wait" is a queue worker, which is integration pattern 2 in the add-an-LLM guide: the queue absorbs bursts of any size, and the workers drain it at exactly the provisioned rate.

One pool, many features: fairness

The limit is per account (or per workspace/key, provider-dependent), and that creates the noisy-neighbor problem you know from shared databases: the nightly backfill job and the customer-facing chat share one TPM pool, and the backfill neither knows nor cares that it's starving checkout.

The fix is the same one you use for pools: sub-budgets by priority, enforced at the chokepoint. Interactive features get a guaranteed slice and shed batch traffic first; batch jobs get throttled to whatever the interactive tier isn't using. This only works if every call flows through one place that can see and enforce the budgets, which is the gateway seam the next lesson builds. Per-tenant fairness is the same mechanism one level down (a tenant field on the bucket key), and it's how one enthusiastic customer's usage stops being everyone's incident. Stripe's classic rate-limiter writeup covers the priority-and-shedding patterns; they transplant to LLM capacity unchanged.

The capacity-planning arithmetic

Do this math at design time, before launch, per feature. It's four numbers:

1. Peak demand: expected requests/minute at peak (say, 300).
2. Tokens per request: input plus real output, measured from staging traffic, not guessed (say, 2,500; count, don't estimate).
3. Required TPM: 300 × 2,500 = 750k tokens/minute at peak.
4. Compare against your tier's TPM at ~70-80% target utilization: 750k needed vs, say, a 1M limit = 94% at peak. Too hot; request the raise now.

Key takeaways

• RPM and TPM are independent meters and either can 429 you: many small calls trip RPM, a few fat calls trip TPM. Know which one your workload actually stresses.
• Providers admit requests against an estimate that typically includes full max_tokens: oversized output caps silently reserve capacity you never use. Size max_tokens to the task.
• The rate-limit headers on every response are free observability. Log them, graph remaining/limit, alert under ~20% headroom, and tier raises become planned instead of emergencies.
• Throttle client-side with a token bucket sized to ~80% of your limits, admitting on the provider's estimate and reconciling with actual usage. Waiting beats erroring; queues absorb what waiting can't.
• One account limit shared by all features is a noisy-neighbor problem: enforce per-feature (and per-tenant) sub-budgets at the gateway chokepoint, interactive guaranteed, batch shed first.
• Capacity-plan with four numbers (peak rpm × tokens/call vs tier TPM at 70-80% utilization) at design time; the levers in order are smooth, prioritize, spend less, then buy more.