SSE, WebSockets, and queues are the three architecture choices for streaming LLM responses to a browser or a native client, and the choice you make is the one that bites at 100 concurrent users. Streaming hides the latency of token-by-token generation from the reader; the architecture decides whether the backend stays standing when the streams pile up.

The trap is that LLM streaming looks like an extension of the request/response model engineers already know. A handler returns a response; the response simply happens to take a long time and arrive piece by piece. The reality is the opposite. Streaming inverts the lifetime model of an HTTP request, from milliseconds-long and stateless to seconds-or-minutes-long and tightly coupled to a single worker. Issue 001 covered the latency stages inside an LLM call; this issue covers what happens to the backend that has to hold the connection open while those tokens arrive.

This issue walks through the three architectural patterns that dominate production LLM streaming, the failure modes that show up at scale, the reconnect problem and three ways to solve it, and a production checklist for the backend that has to survive a thousand concurrent streams.

Why streaming breaks under load

A traditional HTTP handler holds a connection for milliseconds: parse the request, do some work, return the response, close the socket. A single worker can serve hundreds of such requests per second. A streaming LLM handler holds a connection for the full duration of a response, which is typically two to thirty seconds for a chat completion and can stretch to several minutes for a long-context generation. Three failure modes follow from that single shift.

The first failure mode is that the worker pool blocks. A synchronous worker model (gunicorn with the default sync worker, an older Flask deployment, any pre-fork model that assigns one worker per connection for the connection's lifetime) cannot keep up. With sixteen workers and a thirty-second average stream, the pool serves roughly thirty streams per minute. Add a hundred concurrent users and most requests sit in the accept queue waiting for a worker that will not free up for half a minute. The fix is not more workers; it is a different worker model.

The second failure mode is that proxies buffer the response. The default nginx configuration buffers proxy responses in memory before flushing to the client (the initial buffer is one memory page, typically 4 or 8 KB, and the full buffer pool grows into tens of kilobytes). For a streaming response, that buffer holds the tokens until either it fills or the upstream closes, so the browser sees nothing for several seconds and the streaming user experience evaporates. Cloudflare, Google Cloud Load Balancer, and AWS Application Load Balancer all have analogous defaults; each needs an explicit configuration change to pass the stream through without buffering.

The third failure mode is that the load balancer disconnects. AWS ALB has a default idle timeout of sixty seconds. Cloudflare's default is one hundred seconds. A long generation that runs past the timeout is killed mid-stream from the client's perspective, even though the upstream LLM call is still producing tokens. Either the timeout is raised to cover the maximum expected stream duration, or the architecture is designed to survive mid-stream disconnects so the timeout becomes a non-issue.

The combined effect is a backend that handles a single streaming user fine in development, struggles at ten concurrent users, and falls over at a hundred. The architectural choice ahead of that wall determines whether the fix is a config change, a worker-model swap, or a rewrite.

The three architectural patterns

Three patterns dominate production LLM streaming. Each makes a different tradeoff between simplicity, reconnect-friendliness, and operational cost.

Server-Sent Events (SSE): SSE is an HTTP-based, one-way streaming protocol from server to client. The server holds the connection open and writes events as text chunks separated by blank lines. The browser's native EventSource API consumes the stream, and the protocol includes a built-in reconnect mechanism: when the connection drops, the browser reconnects automatically and sends a Last-Event-ID header so the server can resume from where it left off. SSE works over standard HTTP, plays well with existing authentication and middleware, and is the simplest of the three patterns to implement. The cost is that the server must hold the connection for the full duration of the response, which couples client lifetime to worker lifetime unless the architecture takes care to decouple them.

WebSockets: WebSockets establish a persistent, two-way connection over a single TCP socket after an HTTP upgrade. The protocol is symmetric: client and server can each send messages at any time. For LLM streaming, the two-way capability is rarely necessary (the client sends a prompt; the server returns tokens), so WebSockets often add complexity without a corresponding benefit. The protocol also does not include a built-in reconnect or replay mechanism, so any mid-stream disconnect requires application-level logic to resume. WebSockets are the right answer when the application genuinely needs bidirectional communication during the stream (for example, interrupting the generation with a follow-up instruction), and they are the wrong answer when SSE would suffice.

Queue-based (decoupled): The queue-based pattern separates the generation lifecycle from the delivery lifecycle. The client sends a request, the API server enqueues a job and returns a job identifier immediately, and a separate worker consumes the job, generates tokens, and writes them to a stream-backed store (Redis Streams, Kafka, PostgreSQL with LISTEN/NOTIFY, or similar). The client then connects to a streaming endpoint with the job identifier and consumes whatever tokens have been produced. The pattern is more complex than SSE or WebSockets but the most resilient: the worker producing the tokens is decoupled from the client receiving them, so a client disconnect does not cancel the generation and a reconnect simply resumes from the queue.

The diagram above shows the three patterns side by side. SSE and WebSockets are coupled architectures: the same worker that generates the tokens holds the client connection. The queue-based pattern is decoupled: one process generates, another delivers, and a persistent store sits between them. The decoupling is exactly what makes the queue-based pattern survive client reconnects and worker restarts without losing progress.

The reconnect problem

A streaming connection that lives for thirty seconds will, on a real network, drop a meaningful percentage of the time. Mobile clients are the worst case (cellular handoff, brief loss of signal, the screen locking and killing the radio), but desktop clients on Wi-Fi disconnect often enough to matter. A streaming architecture that does not handle reconnects degrades to a worse user experience than non-streaming would have provided: the user sees the first half of the response, then nothing, and assumes the system is broken.

The three architectures handle reconnects differently.

SSE has the cleanest story. The EventSource API automatically reconnects when the connection drops. Each event the server emits carries an id: field, and on reconnect the browser sends Last-Event-ID as a request header. The server is expected to use that header to resume from the correct point in the response. The implementation cost is that the server must either hold a partial response in memory keyed by event id (acceptable for short responses) or write tokens to a persistent store and read from there on reconnect (necessary for long responses or for resilience to worker restarts).

WebSockets require application-level logic. The client must detect a closed socket, retry the connection with the original request context, and either restart the generation from the beginning (wasteful and visible to the user) or send a resume marker the server uses to skip already-delivered tokens. Most implementations skip this work and accept the visible failure, which is one of the reasons the Vercel AI SDK, the Anthropic and OpenAI client libraries, and most published reference implementations default to SSE when the use case is one-way streaming.

The queue-based pattern makes reconnects trivial in the client and the server. The tokens are already in the stream store; the client reconnects to the streaming endpoint with the job identifier and reads from wherever it left off, using Last-Event-ID if the delivery layer is SSE-based. The complexity moves to the queue infrastructure, but once that infrastructure exists, reconnect handling is a single read offset.

The diagram shows how the queue-based pattern composes with SSE to handle reconnects. The worker continues to produce tokens regardless of client connection state; the client resumes by passing the last event id it received. Neither the worker nor the queue cares that the client disconnected.

The cancellation problem

The reconnect problem has a mirror. When the client genuinely goes away (closes the tab, navigates elsewhere, kills the app), the server should stop generating. An LLM token stream costs money for every token produced, and a worker that keeps generating into a dead connection burns the budget for no benefit.

In the coupled architectures (SSE, WebSockets), cancellation is straightforward. The connection close is the signal, and the request handler can propagate cancellation through to the LLM client. Anthropic's SDK and the OpenAI SDK both support cancellation by aborting the underlying HTTP request, which propagates through to the model provider.

In the queue-based architecture, cancellation requires explicit work. The worker is decoupled from the client by design, so it has no direct signal that the client has gone away. Two patterns address this. The first is a watchdog: the streaming endpoint records a "last read" timestamp on every read, and a background process cancels jobs whose last-read timestamp is older than a threshold (typically thirty to sixty seconds). The second is explicit client signalling: the client sends a DELETE /jobs/{id} before navigating away. Both patterns waste some tokens on edge cases (the watchdog runs once a minute, so up to a minute of tokens is paid for), but they bound the cost.

How to pick

Three properties of the application decide the architectural choice.

The first is reconnect sensitivity. If a meaningful fraction of clients are mobile or on flaky networks, the architecture must survive mid-stream disconnects without restarting the generation. SSE with a queue-backed event store is the minimum; the fully queue-based pattern is the strongest answer.

The second is bidirectional need. If the application genuinely needs to send messages to the client mid-stream that are not part of the LLM response (interrupting the generation, sending tool-call decisions, updating progress against a multi-step plan), WebSockets carry their weight. If the only thing flowing during the stream is the LLM's tokens, SSE is simpler and almost always sufficient.

The third is operational tolerance for the queue infrastructure. SSE on a synchronous worker model breaks at ten concurrent users. SSE on async workers (Starlette, FastAPI on Uvicorn, ASP.NET Core with IAsyncEnumerable, Node native streams) holds up at hundreds to thousands. The queue-based pattern survives essentially any concurrent load because the slow part (generation) and the bursty part (client connections) are separated, but the operational cost of running Redis Streams, Kafka, or an equivalent is the price of that survival.

Common mistakes

Four mistakes recur in production LLM streaming implementations.

The first is shipping streaming on a synchronous worker model. Gunicorn's default sync worker, or any pre-fork model that assigns one worker per request for the request's lifetime, cannot scale streaming responses past tens of concurrent users. The fix is async workers (Uvicorn, Hypercorn, ASGI in general) or a queue-based architecture, not more sync workers.

The second is forgetting proxy buffer configuration. nginx, Cloudflare, AWS ALB, and similar proxies default to buffering responses, which destroys the streaming user experience. nginx needs proxy_buffering off and the X-Accel-Buffering: no response header. Cloudflare switches the response to passthrough mode when it sees Content-Type: text/event-stream, so the primary requirement is emitting the right content type, with Cache-Control: no-transform added as belt-and-braces to prevent auto-minification on the same response. Every proxy in the path has to be configured.

The third is leaving load-balancer idle timeouts at their defaults. The default sixty-second timeout on AWS ALB kills the median long-context generation. Either raise the timeout to cover the maximum expected stream duration or, preferably, architect the system to survive disconnects so the timeout becomes a non-issue.

The fourth is forgetting cancellation. A handler that does not propagate the client disconnect into a cancellation of the LLM call generates tokens that nobody reads. At any non-trivial scale, the cost compounds. Wire the disconnect into the LLM client's cancellation token, or in the queue-based pattern, run a watchdog that kills jobs with no recent reader.

Summary

LLM streaming inverts the lifetime model of an HTTP request, and the architectural choice for streaming decides whether the backend survives a hundred concurrent users. SSE is the right default for one-way streaming on async workers, with a queue-backed event store added as soon as reconnect resilience matters. WebSockets are the right answer when the application genuinely needs bidirectional communication during the stream. The queue-based pattern is the strongest answer for applications where mid-stream reliability is the binding constraint, at the cost of running queue infrastructure. The mistakes that break streaming under load are knowable and fixable: sync worker models, proxy buffering, low load-balancer timeouts, and missing cancellation. Treat the choice as load-bearing for every surface that depends on it.

Production checklist

  • Run streaming endpoints on an async worker model (Uvicorn, Hypercorn, Node native streams, or equivalent). Do not run streaming on synchronous pre-fork workers.

  • Disable proxy buffering at every layer in the path: nginx proxy_buffering off, the X-Accel-Buffering: no response header, and the equivalent for Cloudflare, AWS ALB, and any other proxy between the worker and the client.

  • Raise the load-balancer idle timeout to cover the maximum expected stream duration, or architect the system to survive mid-stream disconnects so the timeout is not the binding constraint.

  • Use SSE with id: fields on every event so the browser's EventSource can reconnect with Last-Event-ID. The server reads the header on reconnect and resumes from the correct offset.

  • Propagate client disconnect into LLM cancellation. Wire the request's cancellation token through to the model SDK's abort signal so a closed connection stops generation immediately.

  • For applications with high mobile traffic or stringent reconnect requirements, adopt the queue-based pattern: enqueue the job synchronously, generate asynchronously into a stream store, and deliver via SSE keyed on the job identifier.

  • In the queue-based architecture, run a watchdog that cancels jobs whose last-read timestamp is older than thirty to sixty seconds, so abandoned streams stop costing tokens.

  • Track per-stream metrics separately from per-request metrics: stream duration, tokens delivered, reconnects per stream, and the fraction of streams that complete versus disconnect. The aggregate hides the patterns that matter.

Further reading