How to Stream LLM Responses: SSE vs WebSockets (and the Fetch-Stream Option)

Short answer: stream LLM tokens over SSE or a plain streamed HTTP response, and reach for WebSockets only if you genuinely need bidirectional traffic mid-generation. LLM output is a one-way flow of small text deltas (precisely the workload server-sent events were designed for), and one-way-over-plain-HTTP is the property that makes everything else (load balancers, auth middleware, proxies, HTTP/2, observability) keep working unchanged.

The real engineering isn't picking the transport anyway. It's the relay: client → your backend → provider, with your backend holding the key, enforcing timeouts, and metering cost. This guide builds that relay, walks the three transport options, and covers the proxy failure modes that make streaming look broken in staging.

Why a relay at all

Never let a browser or mobile app talk to the provider directly. The key ships to the client (public within hours: key hygiene lesson), and every call bypasses your rate limits, logging, and spend caps. The relay is non-negotiable; it's also where all the transport decisions live. Latency isn't a concern: your hop adds single-digit milliseconds against a stream whose time-to-first-token is 200–800ms.

Option 1: SSE, the default

SSE is plain HTTP with Content-Type: text/event-stream and a trivial wire format (data: ...\n\n frames). An Express relay against Anthropic's streaming API:

app.post("/api/chat/stream", async (req, res) => {
  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    Connection: "keep-alive",
    "X-Accel-Buffering": "no",          // tells nginx: do not buffer this
  });
 
  // Client closed the tab? Stop paying for tokens nobody will read.
  const upstream = new AbortController();
  req.on("close", () => upstream.abort());
 
  try {
    const stream = client.messages.stream(
      { model: MODEL, max_tokens: 1024, messages: req.body.messages },
      { signal: upstream.signal }
    );
    let i = 0;
    for await (const delta of stream.textStream) {
      res.write(`id: ${i++}\ndata: ${JSON.stringify({ delta })}\n\n`);
    }
    const final = await stream.finalMessage();
    res.write(`event: done\ndata: ${JSON.stringify(final.usage)}\n\n`); // usage = your billing meter
  } catch (err) {
    if (!upstream.signal.aborted) {
      res.write(`event: error\ndata: ${JSON.stringify({ message: "generation_failed" })}\n\n`);
    }
  }
  res.end();
});

Three details in there that separate demos from production: the abort wiring (a closed tab must cancel the upstream call, or you pay for orphaned generations), the usage frame at the end (log it; it's your billing meter), and the error event (a stream that just stops is indistinguishable from a stall; say why).

One classic SSE limitation is gone in practice: the browser EventSource API only does GET without headers, but nothing requires EventSource: POST with fetch and parse the SSE frames from the response body, which is what every LLM provider's own SDK does. (What those provider-side frames actually contain, per dialect, is the stream-anatomy lesson.)

Use it when a human is watching tokens render: chat, drafting, copilots. That's most LLM streaming.

Option 2: Raw fetch streaming, SSE minus the framing

You don't strictly need SSE framing. A chunked HTTP response whose body you read with a ReadableStream does the same job with less ceremony: write raw deltas, read them with res.body.getReader(). The trade: you lose SSE's built-in conventions (event types, IDs for resume, comment keep-alives) and invent your own micro-protocol the moment you need to signal errors or completion mid-stream. Fine for a single tightly-coupled frontend; SSE's conventions pay for themselves the moment a second consumer appears.

Use it when you control both ends, want minimal ceremony, and one consumer exists.

Option 3: WebSockets, for actual bidirectionality

WebSockets are a stateful protocol upgrade: sticky connections, custom auth (no standard headers after upgrade), separate idle-timeout tuning on every proxy in the chain, and a connection registry if you scale horizontally. None of that buys anything for one-directional token flow. It buys a lot when traffic is genuinely two-way during generation:

• Voice agents: audio up and tokens/audio down, simultaneously.
• Interactive interruption: user can stop or steer generation mid-stream (chat "stop" buttons don't need this; an abort on the SSE connection does).
• You already run WebSocket infrastructure: an existing collaborative app adding LLM tokens as one more message type on an established socket. Don't build a second transport out of principle.

Wrong when it's chosen because "streaming = WebSockets." That instinct predates SSE's ubiquity and buys you the operational bill for nothing.

The decision table

The failure modes: it's always a proxy

Buffering. The #1 "streaming is broken" cause: tokens arrive in one burst after 20 seconds. Something between the model and the browser is buffering: nginx (proxy_buffering off, or per-response X-Accel-Buffering: no), compression middleware (skip gzip for text/event-stream), or a platform that buffers entire responses (many serverless runtimes; check yours before committing to an edge deploy). Debug with curl -N directly against each hop.

Idle timeouts. LLMs can pause several seconds mid-generation (tool calls, long prompts). ALBs and nginx kill "idle" connections around 60s defaults. Fix twice: raise the proxy read timeout for the streaming route, and send SSE comment keep-alives (: ping\n\n) every ~15s from the relay.

Stage-blind timeouts. One flat timeout is wrong at both ends: kills healthy long generations and waits forever on dead connections. Time-box TTFT tightly (~10s) and total duration generously (60–120s), per the latency lesson.

Orphaned generations. Every disconnect path must abort the upstream call. Miss one and you'll find it on the invoice, not in the error logs.

If you remember one thing: the transport is a solved problem (SSE by default, WebSockets for real bidirectionality), and the engineering lives in the relay: abort propagation, keep-alives, stage-aware timeouts, and logging usage on every stream. The track's serving module builds the full production relay.

Sources & further reading: MDN: Server-sent events · Anthropic streaming API · MDN: ReadableStream · nginx proxy_buffering