Crabhaus

What it is

Storing a model's weights in fewer bits - say 4-bit integers instead of 16-bit floats - so the same network needs a fraction of the memory and bandwidth. The file shrinks roughly in proportion to the bit width, at the cost of a tiny approximation error in every weight.

How it works

Weights cluster in a narrow bell-shaped range, so small groups of them are mapped onto a coarse integer grid with one scale factor per group; grouping stops rare outliers from wrecking precision for everyone else. Methods like GPTQ and AWQ calibrate on sample data to minimize each layer's output error, and formats like GGUF's K-quants use non-uniform grids matched to the weight distribution.

A concrete example

A 32B model at 16 bits needs ~64 GB for weights alone - no consumer GPU holds that. Its 4-bit GGUF build lands around 18-20 GB and fits a 24 GB RTX 4090 or a 32 GB MacBook with room for the KV cache. That is why the default file Ollama pulls for most models is a 4-bit variant.

Why it matters

Quantization is what makes local and on-device AI possible, and it cuts cloud serving costs too. Generating tokens is memory-bandwidth-bound - each token streams all the weights past the compute units - so halving the bytes also speeds up decoding, not just storage.