Crabhaus

What it is

An inference technique that makes a large model generate faster without changing its output. A small 'draft' model guesses several tokens ahead; the large 'target' model checks all the guesses in one forward pass and keeps those it agrees with. Accepted guesses are nearly free; any rejection falls back to the big model's own token.

How it works

Decoding is memory-bound: each step streams all weights to emit one token, so verifying K tokens costs about what generating one does. The draft cheaply proposes K tokens; the target scores them in one pass, accepts each with probability min(1, p_target/p_draft), and replaces the first reject from a corrected distribution. This rejection-sampling rule provably preserves the target's output.

A concrete example

Pair a 70B model with a 7B draft at K=5. On boilerplate like 'return response.json()' all five guesses verify in one pass - five tokens for one large-model step. On a subtle prose question, acceptance drops and decoding gracefully degrades to normal speed, never below it in quality. Google's and DeepMind's papers both report roughly 2-3x end to end.

Why it matters

It converts idle GPU compute into extra tokens per second at identical quality - one of the quiet levers behind falling per-token prices. Serving stacks like vLLM ship it as a config flag, and it works locally too: a 1-3B draft can meaningfully speed a 30-70B model on a workstation. It also explains why one endpoint feels fast on boilerplate and slower on unusual text.