Crabhaus

What it is

VRAM is a GPU's on-board memory and the hard budget for running a model locally. A model 'fits' when its weights, its growing context cache, and runtime overhead sit inside that budget at once - so fit depends on your settings, not the parameter count alone.

How it works

Weights cost parameters times bytes each: 2 at FP16, 1 at 8-bit, about 0.5 at 4-bit. Every token in context then adds key/value cache - roughly 2 x layers x KV-heads x head-dim x bytes - so the cache grows linearly with context. Add 1-2GB of overhead and 10-15% headroom; if the total is under your VRAM, it fits.

A concrete example

An 8B model at FP16 needs about 16GB for weights alone, so it misses a 12GB card. Quantized to 4-bit it is about 4.9GB and fits easily - until context bites: its KV cache runs about 128KB per token, so a full 128K context adds roughly 16GB, more than the weights. At 8K-16K context the 12GB card is comfortable.

Why it matters

This one estimate decides which GPU to buy, which quantization to serve, how much context an agent can afford, and when local stops being cheaper than an API. Long-running agents usually blow VRAM through KV-cache growth, not weights - context budgets matter as much as model choice.