Crabhaus

What it is

The stack for running open-weight models on your own hardware. llama.cpp is a C/C++ engine that runs LLMs on ordinary CPUs and consumer GPUs; GGUF packs quantized weights, tokenizer, and metadata into one file it can memory-map; Ollama wraps it in a local server with one-command model pulls and an OpenAI-compatible API on localhost.

How it works

Weights are quantized from 16-bit floats to ~4-5 bits (schemes like Q4_K_M), shrinking an 8B model from ~16GB to ~5GB. Generating each token streams essentially every weight through the processor, so speed is bound by memory bandwidth: tokens/sec is roughly bandwidth divided by model bytes. A GPU takes as many layers as fit in VRAM; the CPU runs the rest. Ollama keeps models warm.

A concrete example

A 16GB laptop moving ~100GB/s pulls a ~5GB 4-bit 8B model and generates near the predicted 100/5 = 20 tokens/sec - fine for chat, drafting, and code completion. A 70B model won't fit: even at 4 bits it needs ~40GB for weights alone, before the KV cache that grows with context length.

Why it matters

Prompts never leave the machine, it works offline, and tokens are free - changing the economics of high-volume small-model jobs like extraction and autocomplete. The bandwidth ceiling tells you when local wins (private, latency-tolerant work) and when cloud does (frontier reasoning, long contexts). It is also why fast unified-memory laptops became the default local-LLM box.