streaming-llm-responses-sse
verified70a3d562-4c0c-49b2-810b-b4b38cc5645e
Stream LLM output to clients with Server-Sent Events ā token streaming, backpressure, partial parsing, cancellation, and breaking a full request into deltas.
Metadata
Skill file
# Streaming LLM Responses with SSE
Use when you want the user to see tokens the moment they're generated (chat apps,
agent UIs, code completion) instead of waiting for a full completion ā or when
responses are long enough that time-to-first-token matters. Server-Sent Events (SSE)
is the simplest robust way to push a stream of deltas from your backend to a browser
or native client.
## Why SSE (and not WebSockets) for LLM output
- **SSE** = one-way, HTTP-based event stream (`text/event-stream`), auto-reconnect,
works over plain HTTP/1.1 and through proxies ā perfect for a *server ā client*
token stream.
- **WebSockets** = bidirectional ā needed only if the client must also stream data
*back* mid-connection (rare for chat). Pick SSE unless you truly need full-duplex.
Typical flow:
```
client --POST /chat {messages}--> backend
backend --200 text/event-stream--> client
event: delta data: {"delta":"Hello"}
event: delta data: {"delta":" world"}
event: done data: {"id":"...","usage":{...}}
```
## Backend pattern (FastAPI + async streaming)
The key is to **stream tokens straight from the provider's generator** to the SSE
response, never buffering the whole completion:
```python
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import json
app = FastAPI()
async def event_stream(messages):
tokens = await llm.chat.completions.create(
model="...", messages=messages, stream=True) # provider stream
async for chunk in tokens:
delta = chunk.choices[0].delta.content
if delta:
yield f"event: delta\ndata: {json.dumps({'delta': delta})}\n\n"
yield f"event: done\ndata: {json.dumps({'status': 'ok'})}\n\n"
@app.post("/chat")
async def chat(req: dict):
return StreamingResponse(event_stream(req["messages"]),
media_type="text/event-stream")
```
Each SSE message is `event: <name>` + `data: <json>` + a blank line. Batch delta
tokens into small arrays to cut event count without hurting perceived latency.
## Client responsibilities
- Take both **time-to-first-token (TTFT)** and **tokens/sec** as the latency metrics;
total completion time is less important for perceived quality.
- Render deltas incrementally; do **not** re-render the whole message each tick
(causes flicker).
- Handle the `done` event to finalize (attach final metadata/usage, stop spinner).
- Use the browser's built-in `EventSource` for GET, or `fetch` with streaming reads for
POST bodies (EventSource only does GET; POST streaming needs `fetch` + reading the
response body stream).
## Backpressure & errors
- **Backpressure:** don't let the producer outpace the consumer unboundedly. If the
client is slow, drop to batching/maybe-throttle deltas rather than buffering
everything in memory.
- **Cancellation:** when the client disconnects, the generator must stop the
upstream LLM call (otherwise you keep paying token cost for a dead client).
Detect `asyncio.CancelledError` / disconnect and cancel the provider request.
- **Mid-stream errors:** send an `event: error` with the message so the UI can show
a graceful partial-response prompt instead of a blank hang.
- **Retries:** client-side resumability of a *finished* stream is limited; for long
runs consider persisting partial output server-side so a reconnect can continue.
## Pitfalls
- **Proxy buffering:** nginx/proxies may buffer SSE and defeat streaming ā disable
buffering (`proxy_buffering off;`) and ensure `X-Accel-Buffering: no`.
- **gzip/compression delaying deltas:** compression can add TTFT latency ā weigh
enabling compression for SSE carefully.
- **Forgetting the blank-line terminator** between SSE events breaks the protocol.
- **Not flushing:** some servers buffer output on aiohttp/gunicorn ā ensure the
stream is flushed per event.
- **Sending full JSON per token:** huge overhead; send only the delta plus event type.
- **No cancellation** ā burning tokens on abandoned requests (cost + capacity).
## Verify
- TTFT is near the provider's first token (not the whole completion's).
- A long prompt streams tokens incrementally (measure tokens/sec) and the UI paints
progressively.
- Disconnecting mid-stream stops the upstream call (confirmed in provider logs / token
usage).
- Passing through nginx/reverse proxy preserves streaming (no end-buffering).
- The client correctly finalizes on `done` and surfaces `error` events gracefully.