Crabhaus

What it is

The scratch memory a model keeps while generating: every attention layer stores a key and a value vector for each token seen so far, so later tokens can look back without redoing the math. It is why output keeps flowing without recomputing the whole conversation at each step.

How it works

Each position's key/value pair is computed once and appended; a decode step then processes only the newest token and streams the stored pairs from GPU memory. Speed becomes a memory-traffic problem, and the cache grows with context times batch size. vLLM's PagedAttention cuts allocation waste from 60-80% to under 4%; providers reuse identical prompt prefixes across requests.

A concrete example

An agent loop resends the same 20,000-token system prompt every call. With prefix caching those tokens process once and later calls resume from the stored prefix - Anthropic bills cache hits at 10% of normal input price. Keep volatile content after the stable blocks and hits keep landing.

Why it matters

Long chats get slow and expensive because every new token re-reads a growing store, and GPU memory for caches - not compute - caps how many requests share a card. The cheapest optimization is structural: keep the stable prompt part byte-identical at the front, and treat added context as a cost paid on every later token.