What's Actually in the Stream

There are two legs to every streamed LLM response: the provider to your backend, and your backend to your client. The streaming guide covers the second leg (SSE vs WebSockets, the relay, proxy buffering). This lesson is the first leg: the wire format the provider actually sends you, because that's the layer you'll be staring at when streaming misbehaves in production.

The SDKs hide this behind pleasant iterators, and you should use them. But "the stream just stops sometimes" and "our token dashboards undercount" and "we get charged for answers nobody read" are all first-leg problems, and you can't debug a protocol you've never looked at.

Both streams are SSE; the grammar differs

Under the hood, both providers stream over server-sent events: a long-lived HTTP response emitting data: frames. What differs is the event vocabulary, and the differences are exactly where the bugs live:

{"delta":{"content":"Hel"}}
{"delta":{"content":"lo"}}
{"delta":{},"finish_reason":"stop"}

message_start    input tokens
content_block_start
content_block_delta  "Hel" "lo" …
content_block_stop
message_delta    stop_reason +
                 output tokens
message_stop

Read Anthropic's sequence once more, because it encodes two facts you'll rely on later: input tokens are known at the start (message_start, before any generation) and output tokens and stop_reason are known only at the end (message_delta). OpenAI's shape implies the same thing: the interesting metadata arrives in the final frames. Full event references: Anthropic streaming, OpenAI streaming.

Consuming it in Node

With an SDK, the loop is a plain async iterator. The production version does three jobs the demo skips: aggregate the deltas, capture the final usage, and accept an abort signal (next section):

import Anthropic from "@anthropic-ai/sdk";
 
const client = new Anthropic();
 
export async function streamCompletion(
  params: { system: string; messages: Anthropic.MessageParam[] },
  onDelta: (text: string) => void,          // feed the relay, a job, wherever
  signal?: AbortSignal
) {
  const stream = client.messages.stream(
    { model: MODELS["support-chat"].id, max_tokens: 1024, ...params },
    { signal }
  );
 
  for await (const delta of stream.textStream) {
    onDelta(delta);                          // fragments, not tokens: don't assume 1:1
  }
 
  const final = await stream.finalMessage(); // full aggregated message
  return {
    text: final.content[0].type === "text" ? final.content[0].text : "",
    stopReason: final.stop_reason,           // check it: "max_tokens" means truncated
    usage: final.usage,                      // your billing meter, only complete here
    requestId: final._request_id,
  };
}

The OpenAI SDK is the same pattern with different property names (for await (const chunk of stream), fragments at chunk.choices[0].delta.content). One habit worth keeping from the first API call lesson: a delta is a text fragment, not a token; providers batch tokens into frames however they like, so never count frames and call it a token count.

The three ways a stream ends

Buffered calls fail loudly: you get a status code. Streams have already said 200 OK before anything went wrong, so failure arrives inside the stream, in one of three shapes:

1. The clean stop. Final events arrive, stop_reason/finish_reason is set, usage is complete. Still check the stop reason: max_tokens on a 200 is a truncated answer, which is a failure that arrives as a success.
2. The explicit error. Anthropic can emit an error event mid-stream (an overloaded_error is the classic); OpenAI streams typically just terminate abnormally after the HTTP layer already committed to 200. Either way: the text you aggregated so far is a partial, and whether partials are servable is a product decision, not an exception handler's.
3. The silent stall. No error, no close, no deltas. Nothing in the protocol tells you; only your inter-token stall timer does. That's the third timer in the stage-aware timeout pattern: tight on first token, rolling stall guard between deltas, hard deadline on the whole call.

If HTTP trailers had won, this would all feel familiar: headers up front, body in the middle, the accounting metadata at the end. That's what a stream is; design your logging around it.

Cancellation: stop paying for abandoned answers

A user closes the tab five seconds into a forty-second answer. Nothing about the provider connection knows or cares; the model keeps generating, and you keep paying, to completion. At scale this is a real line item: chat users abandon long answers constantly.

The fix is one signal wired end to end. Whatever tells you the consumer is gone (the client disconnect on your relay, a canceled job, a timeout) must abort the upstream call:

const upstream = new AbortController();
 
// Any "nobody is reading this anymore" signal aborts the provider call:
req.on("close", () => upstream.abort());          // HTTP caller disconnected
job.onCancel(() => upstream.abort());             // or: queue job canceled
 
const result = await streamCompletion(params, sendToClient, upstream.signal);

Aborting closes the connection and the provider stops generating: you pay for tokens generated up to the abort, not the full answer. Treat a missed abort path like a leaked DB cursor: invisible in the error logs, visible on the invoice. The streaming guide calls the same rule "orphaned generations" on the client-facing leg; the point here is that every consumer of streamCompletion, human or machine, needs an abort story.

One boundary note: when tool use enters the picture, both providers add more block types to the stream (tool-call deltas interleaved with text). Same grammar, more vocabulary; module 3 introduces tools and module 5 streams them.

Buffer or stream? Same rule as before

Nothing in this lesson obligates you to stream. The rule from lesson 3 stands: stream anything a human watches, buffer anything a machine parses. A queue worker summarizing tickets should make buffered calls and skip this entire event grammar; a chat endpoint can't. When you do stream and need to push it onward to browsers, that's the second leg, and the guide has the relay, the transport table, and the proxy war stories.

Key takeaways

• Both providers stream SSE, but the grammar differs: OpenAI repeats one chat.completion.chunk shape and ends with [DONE]; Anthropic sends a typed event sequence (message_start through message_stop) with ping keep-alives and possible mid-stream error events.
• The stream's two ends carry the metadata: input tokens at the start, stop_reason and output tokens only at the end. Aggregate deltas as fragments, never as tokens.
• Streams end three ways: clean stop (check stop_reason anyway), explicit mid-stream error (your text is a partial), and silent stall (only your stall timer catches it).
• Usage goes missing on every non-clean ending while the provider still bills: log estimates for aborted and errored streams or your cost dashboards lie precisely during incidents.
• Wire an AbortController from every consumer to the provider call. An abandoned generation with no abort path is money spent on an answer nobody read.
• On OpenAI, remember stream_options: {"include_usage": true}, or the stream never tells you what it cost.