Crabhaus

What it is

Three redesigns of transformer attention that shrink the key-value (KV) cache - the per-token memory a model keeps so it doesn't recompute attention over the whole context at every decoding step. MQA shares one key-value head across all query heads, GQA gives each group of query heads a shared key-value head, and MLA compresses keys and values into one small latent vector and caches that instead.

How it works

Classic multi-head attention stores separate keys and values per head, so cache grows with heads x layers x context, and decoding is limited by streaming it from GPU memory. MQA (2019) shares one K/V set: maximum savings, some quality loss. GQA (2023) interpolates, e.g. 8 KV heads for 64 query heads, near full quality. MLA (DeepSeek-V2) caches one low-rank latent per token, cutting cache 93.3%.

A concrete example

A 128K-token context on a full multi-head model can demand tens of gigabytes of KV cache per request, crowding out concurrent users. GQA with an 8x head reduction stores roughly one-eighth of that; MLA a small fraction - how DeepSeek-V2 serves 128K contexts economically, and why Llama-class models ship GQA by default.

Why it matters

At serving time the KV cache, not the weights, is usually what runs a GPU out of memory - it grows with every concurrent user and every token of context. Shrinking it converts directly into longer contexts, bigger batches, and cheaper tokens, which is why nearly every current frontier and open model uses GQA or MLA.