The API Is Stateless. Your Chat Isn't.

HTTP is stateless, and you've never once been confused by that, because the discipline is muscle memory: the protocol remembers nothing, so sessions live in your store, and every request carries enough identity to rehydrate them. LLM APIs are stateless in exactly the same way. The model does not remember your last call. There is no server-side conversation object, no session affinity, nothing.

Every "chat" you've ever used is a client-side illusion: on each turn, the application resends the entire conversation so far as an array of role-tagged messages, and the model predicts what comes next. Which means the moment you ship a multi-turn feature, you own three pieces of infrastructure the demo skipped: a transcript store, a token budget, and a truncation strategy. This lesson builds all three.

One turn, end to end

Here's what "the user sends a message" actually costs your backend:

Two consequences fall straight out of the replay. First, context-window pressure: history grows linearly per turn and the window is a hard cap, so unbounded transcripts eventually overflow it. Second, cost: you re-buy the entire history as input tokens on every single turn, which is how a 30-turn conversation quietly costs 30x its first turn. The latency and cost lesson called this "what silently rides along on every call"; conversation history is the biggest rider.

The transcript store: an event log you replay

The natural schema is the one you'd design for any append-only event stream. Two tables, no surprises:

CREATE TABLE conversations (
  id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id     UUID NOT NULL REFERENCES users(id),
  feature     TEXT NOT NULL,            -- "support-chat", "sql-copilot"
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);
 
CREATE TABLE messages (
  id              BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  conversation_id UUID NOT NULL REFERENCES conversations(id),
  role            TEXT NOT NULL CHECK (role IN ('user', 'assistant')),
  content         TEXT NOT NULL,
  token_count     INT NOT NULL,         -- counted once at write time
  model           TEXT,                 -- assistant rows: which snapshot answered
  request_id      TEXT,                 -- assistant rows: provider trace handle
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);
 
CREATE INDEX ON messages (conversation_id, id);

The non-obvious column is token_count, and it's the one that makes the rest of this lesson cheap. For assistant messages, the count is free: the API's usage block tells you exactly what the response cost. For user messages, count once at insert and store it. Now trimming history to a budget is integer arithmetic over rows you already have, with no tokenizer in the request hot path.

Also worth noticing what the schema stores on assistant rows: the model snapshot and request_id, per the versioning lesson. A transcript is an audit log; six weeks from now, "which model said this to which customer" is a question someone will ask.

Trimming: fit the budget without losing the plot

With per-row token counts stored, the budget enforcer is short. The shape matters more than the code: the system prompt is pinned, the newest turns survive, and history drops from the oldest end in whole user/assistant pairs (an orphaned half-exchange confuses models and violates some providers' alternation rules):

type Msg = { role: "user" | "assistant"; content: string; tokenCount: number };
 
// History gets what's left after the fixed costs are reserved.
export function trimToBudget(history: Msg[], budget: number): Msg[] {
  const kept: Msg[] = [];
  let used = 0;
  // Walk newest-first, admit whole pairs, stop at the budget.
  for (let i = history.length - 1; i >= 0; ) {
    const pair = history[i - 1]?.role === "user" ? [history[i - 1], history[i]] : [history[i]];
    const cost = pair.reduce((sum, m) => sum + m.tokenCount, 0);
    if (used + cost > budget) break;
    kept.unshift(...pair);
    used += cost;
    i -= pair.length;
  }
  return kept;
}
 
// assemble: [system prompt] + trimToBudget(history, HISTORY_BUDGET) + [newMessage]

What number goes in budget? Work backward from the window, the way the context-budget table did: window minus system prompt, minus tool schemas, minus the output reserve, minus headroom. History gets the remainder, as an explicit constant in code review, not an emergent property discovered in an incident.

Sliding window vs. summarization

Dropping old turns is the default strategy. Its failure mode is real, though: the model forgets what fell off the window, and users notice ("I told you my order number three messages ago"). The alternative spends tokens to compress instead of drop:

A hybrid covers most real products: sliding window for the recent tail, one summary message holding the compressed past, and anything that must never be forgotten (the user's plan, the order being discussed) injected as structured fields in the system prompt rather than trusted to survive summarization. Making compaction cheap (when to run it, which model to run it on, caching the prefix) is cost engineering, and module 7 picks that up.

The provider will offer to hold the state. Own it anyway.

OpenAI's Responses API can persist conversation state server-side: pass store: true and chain calls with previous_response_id instead of resending history. Anthropic's Messages API stays fully client-managed; you always send the whole transcript.

Handy for prototypes. For production, owning the transcript is still the right default, for reasons that have nothing to do with the model:

• Portability. Server-side state is the stickiest possible lock-in: your conversation history lives inside one vendor's opaque store, priced and truncated by their rules. The provider-abstraction lesson only works if the state is yours to replay elsewhere.
• Control. Your trimming policy, your retention policy, your deletion guarantees, your audit queries. "How does the provider truncate when the thread outgrows the window" should not be a question you answer with "we're not sure."
• It's your product data. Transcripts feed your evals, your fine-tuning corpus someday, your "what do users actually ask" analytics. Handing the primary copy to a vendor is like letting your payment processor keep the only record of your orders.

Key takeaways

• The API remembers nothing between calls. Every turn resends the full transcript; the "conversation" is state you own, exactly like sessions over stateless HTTP.
• That replay has two bills: history re-purchased as input tokens on every turn, and linear growth toward the context-window cap. Budget for both on turn one, not turn thirty.
• Store transcripts as an append-only event log with a token_count per row (free for assistant messages via usage, counted once for user messages), plus the serving model and request_id on assistant rows for auditability.
• Trim with the system prompt pinned, newest turns kept, oldest dropped in whole user/assistant pairs, against an explicit history budget derived from the window.
• Sliding window for short sessions; add summarization for long ones; keep must-not-forget facts in structured state, never trusted to a lossy summary.
• Provider-managed conversation state (OpenAI Responses) is a convenience, not an architecture: keep the source of truth in your own store for portability, control, and compliance.