Streaming / SSE

What streaming changes

A traditional API response says: do all the work, then return the result. A streaming API changes the contract to return useful pieces as soon as they are ready. This matters when the complete operation takes seconds or minutes but the user can benefit from partial progress immediately.

AI text generation is a common example. Instead of waiting for the full completion, the server can send tokens or chunks as the model generates them. This reduces perceived latency even when total completion time remains unchanged.

Server-Sent Events

SSE is a simple HTTP mechanism for one-way server-to-client streaming. The browser opens a long-lived connection, commonly requesting text/event-stream, and the server emits a sequence of events. Each event can include data, an event name, an identifier, and retry information.

SSE is unidirectional. The server pushes events to the client; the client generally sends separate HTTP requests when it needs to provide additional input. This is different from WebSockets, which provide bidirectional messaging over a persistent connection.

Why SSE works well for AI interfaces

SSE fits naturally into browser-facing HTTP infrastructure. A chat application can keep the normal HTTPS request model while incrementally sending generated text. There is less protocol machinery than a full bidirectional socket, and reconnection behavior is straightforward to implement.

The trade-off is that streaming connections live much longer than ordinary API requests. Reverse proxies can buffer chunks, idle timeouts can terminate quiet connections, and infrastructure must handle cancellation when users navigate away or stop generation.

Operational concerns

A streaming service should define heartbeat behavior, flush semantics, maximum connection duration, cancellation propagation, and reconnect behavior. Gateways and load balancers must be configured so they do not buffer the output that the application intends to stream.

For generated content, the client also needs to treat the stream as a sequence that may end unexpectedly. It should know how to reconstruct partial state and distinguish a clean completion from a network disconnect.

Example

A browser sends a prompt to /api/chat. The backend calls a model server, receives generated chunks, and forwards them over SSE. The UI appends each data: event to the visible answer. If the user presses Stop, the browser closes the stream and the server propagates cancellation to the model worker.

The core idea

Streaming is fundamentally about time-to-first-useful-result, while SSE is one practical HTTP transport for doing it from server to client.