Crabhaus

What it is

Running model inference on the phone, laptop, or embedded chip itself instead of calling a cloud API. A small language, vision, or speech model executes on the device's own CPU, GPU, or NPU, so the prompt and the data it touches never have to leave the hardware in your hand.

How it works

Three moves make a model fit. Compression: weights quantized to ~4 bits or less - Apple's ~3B on-device model averages 3.7 bits per weight. Specialization: lightweight LoRA adapters swapped onto one small base model per task. Hardware mapping: runtimes like Core ML, LiteRT, ONNX Runtime, or llama.cpp compile the math onto the NPU. Prompts too hard for the local model escalate to the cloud.

A concrete example

Apple Intelligence runs a ~3 billion parameter model on the iPhone for writing tools and summaries, swapping LoRA adapters per feature and handing harder requests to a server model under Private Cloud Compute. The DIY version: llama.cpp runs quantized open-weights GGUF models on a laptop with no network at all.

Why it matters

Privacy: data is processed where it lives. Latency: no network round trip, so responses start in milliseconds. Cost: no per-token API bill, which matters when AI runs on every keystroke or camera frame. Offline: features keep working without connectivity. And it is reshaping hardware demand, from NPUs in every laptop SoC to edge boards.