Tokens, Context Windows, and Your New Resource Limits

Every resource limit you've ever tuned (payload caps, connection pools, RAM) has an LLM equivalent, and they're all denominated in the same unit: the token. Tokens are simultaneously your billing meter, your latency driver, and your memory ceiling. If you can budget bytes, you can budget tokens; you just need the conversion factors.

Tokens: the byte of LLM systems

Models don't read characters. Input text is split by a tokenizer into tokens: subword chunks from a fixed vocabulary (typically 50k–200k entries). "unbelievable" might become un + believ + able. Common words are one token; rare words, code identifiers, and non-English text shatter into more.

Why you care: every quota that matters is counted in tokens.

• Billing is per token, priced separately for input and output.
• Latency scales with tokens: input tokens set time-to-first-token, output tokens dominate total time (generated one at a time, typically 50–200 tokens/second as of mid-2026; the full latency model is in the latency and cost lesson).
• Memory (the context window) is a token count, not a byte count.

Rules of thumb

For English prose, ~4 characters or ~0.75 words per token. Everything else is denser or worse. Estimates that are wrong by 2x will wreck a cost model, so internalize this table:

Two practical consequences. First, minify JSON before sending it to a model: pretty-printing can cost 30–50% more tokens for zero benefit. Second, never eyeball token counts for anything that hits a budget or a bill; count them (code below).

The context window: payload limit plus RAM

The context window is the maximum number of tokens a model can process in one call, input and output combined. As of mid-2026, mainstream models sit at 128k–200k tokens, with long-context variants at 1M+. It behaves like two limits you already know, fused:

• A request payload cap. Exceed it and the call fails (or, worse, the client silently truncates). Like nginx's client_max_body_size, except your "body" includes conversation history that grows every turn.
• Working memory. Whatever isn't in the window doesn't exist for this call. There's no swap, no page-in. If the answer to the user's question is in turn 3 and you truncated turn 3, the model cannot recall it. It will do the next worst thing: guess plausibly.

What actually consumes the window

In a toy demo, the prompt is the user's question. In a production feature, the user's question is a rounding error. A realistic RAG-backed support assistant on a 128k-token model:

Fine today. But history grows per turn, and a long ticket thread or a big tool result (one verbose API response pasted into context can be 10k+ tokens) eats the rest fast. Budget context like you'd budget a connection pool: reserve capacity per component, enforce caps, and decide the eviction policy (truncate oldest turns? summarize them?) before you hit the ceiling in production. The conversation-state lesson builds the store and the trimmer that enforce it.

Long context is not reliable recall

A 1M-token window does not mean 1M tokens of dependable memory. Two failure modes, both well-documented:

Lost in the middle. Retrieval accuracy is U-shaped over position (Liu et al., 2023): models are strong on information near the start and end of the context and measurably weaker on the middle. Placement is a real lever: put critical instructions and the most relevant retrieved chunks near the top or bottom, not buried at position 60k.

Context rot. As total context grows, performance degrades even on tasks the model aces at 5k tokens (Chroma's context-rot study measured this across 18 models). Irrelevant filler actively hurts: it dilutes attention and gives the model more surface to latch onto the wrong thing. Think of it like a table with no index: the data is technically all "in there," but lookups get slower and flakier as it grows.

The engineering posture this implies: curate, don't dump. Retrieving 5 highly relevant chunks beats stuffing 50 mediocre ones, even when the 50 fit. The window limit is a hard cap; the useful window is smaller and shrinks with noise.

Pricing math: do it before you build

Providers price input and output separately, in USD per million tokens (MTok). As of mid-2026, ballpark ranges:

Output costs 3–5x input. Cached input (a repeated prefix like your system prompt and tool schemas, via prompt caching) is typically discounted ~90%. These two facts shape most cost optimizations.

Worked example: support-ticket summarizer

Feature: summarize every resolved ticket into a 3-sentence digest plus tags. Volume: 10,000 tickets/month.

Per ticket:

• Input: system prompt 400 tokens + ticket thread ~2,600 tokens = 3,000 input tokens
• Output: summary + tags = 200 output tokens

Monthly volume: 30M input tokens, 2M output tokens.

Three lessons hiding in that table. The model choice is a 13x cost spread: run an eval and pick the cheapest model that passes, exactly like right-sizing an instance type. Input dominates here (15x more input than output), so trimming the ticket thread (strip signatures, quoted replies, HTML) pays directly. And at 10k tickets/month even the frontier model is cheap; the same math at 10M events/month is $160k/mo, which is why per-unit token math belongs in the design doc, not the postmortem.

Counting tokens in code

Never estimate when you can count. OpenAI-family tokenizers are available offline via tiktoken (Python) or js-tiktoken (TypeScript); Anthropic and others expose token-counting endpoints. A budget guard in TypeScript:

import { encodingForModel } from "js-tiktoken";
 
const enc = encodingForModel("gpt-4o"); // pick your model's encoding
 
export function countTokens(text: string): number {
  return enc.encode(text).length;
}
 
// Enforce a per-component budget before the API call, not after the bill.
const HISTORY_BUDGET = 20_000;
 
export function fitHistory(turns: string[]): string[] {
  const kept: string[] = [];
  let used = 0;
  for (const turn of [...turns].reverse()) {   // keep most recent first
    const cost = countTokens(turn);
    if (used + cost > HISTORY_BUDGET) break;
    kept.unshift(turn);
    used += cost;
  }
  return kept;
}

Key takeaways

• Tokens are the single unit behind billing, latency, and memory. English is ~4 chars/token; JSON and code are denser; minify JSON and keep base64/hashes out of prompts.
• The context window is a payload cap fused with working memory: exceed it and calls fail; omit something and the model guesses instead of recalling.
• Budget the window per component (system prompt, tool schemas, RAG chunks, history, output reserve) with explicit caps and an eviction policy, like a connection pool.
• Long context degrades: lost-in-the-middle and context rot mean the useful window is smaller than the advertised one. Curate context; don't dump.
• Output tokens cost 3–5x input; cached input is ~90% off. Do per-unit cost math (tokens/request x volume x price) at design time: model choice alone is often a 10x+ spread.
• Count tokens in code (js-tiktoken or provider endpoints) and log actual usage from API responses as your billing ground truth.