Crabhaus

What it is

Every LLM response happens in two phases. Prefill ingests your whole prompt at once, running it in parallel to build the KV cache - the stored attention state for every prompt token. Decode then generates the reply one token at a time, each step reusing that cache. The wait before the first word is prefill; the streaming speed after it is decode.

How it works

Prefill is compute-bound: thousands of prompt tokens processed in one batch of big matrix multiplications keeps the GPU saturated. Decode is memory-bound: each step computes one token but must read the whole KV cache and the model weights from GPU memory, so the chip mostly moves bytes. That is why time-to-first-token grows with prompt length while tokens-per-second stays roughly flat.

A concrete example

Paste a 30-page contract and ask for a two-line summary. The multi-second pause is prefill chewing through ~20,000 tokens; once the first word lands, the answer streams as fast as a one-liner would. Ask a follow-up in the same chat and the pause shrinks - prompt caching lets the provider skip re-prefilling the prefix it already processed.

Why it matters

The split explains pricing and latency that otherwise look arbitrary: input tokens cost less than output because parallel prefill extracts more work per GPU-second than sequential decode, and cache discounts exist because a hit skips prefill entirely. It also gives builders two levers: trim or cache prompts to cut time-to-first-token, cap output length to cut generation time.