Crabhaus

What it is

Batching is how an inference server groups many users' requests so one GPU pass does work for all of them at once. Throughput is how many tokens the system produces per second across everyone; latency is how long any single user waits for their own tokens. Serving is the art of trading these two against each other.

How it works

Token generation is limited by streaming weights from memory, not math; a lone request wastes the GPU. Batching reads the weights once per step for every sequence. Static batching holds a group until its longest member finishes, idling done slots; continuous batching (Orca's iteration-level scheduling) refills them each token step. Bigger batches raise total tokens/sec as each user's stream slows.

A concrete example

One user wants a two-line summary, another a 900-token essay. Static batching parks the summary's finished slot until the essay ends; continuous batching refills it on the next step. Anyscale's benchmark measured up to 23x higher throughput with vLLM's continuous batching versus naive request-level batching, with p50 latency improving too.

Why it matters

Cost per token is throughput inverted: double the tokens per second from the same GPU and each token costs half as much to serve. That is why bulk queues are cheap and strict-latency interactive products pay more. Self-hosting, the latency-bound vs throughput-bound question decides your engine settings and hardware bill.