Crabhaus

What it is

Token streaming is how an LLM API sends its answer incrementally as it is generated, usually as server-sent events (SSE) over one held-open HTTP response, instead of returning the finished text in a single payload. You set stream: true and consume a sequence of small typed events as the model writes.

How it works

The server replies with Content-Type: text/event-stream and writes data: chunks as the model decodes. Two clocks govern the feel: time-to-first-token (TTFT), set by prefill, the compute spent reading your prompt before anything can be emitted, plus queueing; and decode rate (tokens per second), which sets how fast text accrues after that.

A concrete example

A chat app renders each text delta as it lands: first words appear within a few hundred milliseconds, even though the full 800-token answer takes another 20 seconds. Without streaming the user watches a spinner for all 20, and long generations can trip idle timeouts in proxies along the way.

Why it matters

Users judge an AI product by how fast it starts answering, not how fast it finishes, so TTFT is the latency number that moves perceived quality. It is also the lever you control: shortening or caching the prompt cuts prefill, while total time is mostly fixed by output length. Voice agents feel it hardest, since TTFT compounds into time-to-first-audio.