Crabhaus

What it is

The gap between what people expect from software (same input, same output) and what LLMs do. Two separate things break reproducibility: deliberate sampling randomness in picking the next token, and numerical wobble in how inference servers execute the math. Turning off the first does not remove the second, so identical prompts can still return different answers.

How it works

Temperature 0 means always take the top token, which should be deterministic. But servers batch concurrent requests to keep GPUs busy, and batch size shifts with load. Most GPU kernels are not batch-invariant: the order they add floating-point numbers depends on batch size, and float addition is not associative. Tiny logit differences sometimes flip the top token, and the output diverges.

A concrete example

Thinking Machines ran one prompt 1,000 times through Qwen3-235B at temperature 0 and got 80 different completions. After rewriting the inference kernels to be batch-invariant, so reduction order no longer depended on how many requests shared the batch, all 1,000 completions came back bitwise identical.

Why it matters

Run-to-run variance is noise you must engineer around: a prompt tweak that 'improves' an eval by two points may be within natural variance, and you cannot debug an agent failure you cannot replay. Pin seeds and model versions for the sampling layer; use batch-invariant inference when bitwise repeatability is worth the throughput cost.