Crabhaus

What it is

Training a small 'student' model to reproduce a larger 'teacher' model's behavior, using the teacher's own outputs as the training signal. The student never sees the teacher's weights - only what it says.

How it works

Classic distillation trains the student to match the teacher's full output probability distribution, so it learns how the teacher weighs every alternative, not just the right answer. The modern LLM form is simpler: prompt the teacher at scale, collect its responses or reasoning traces, and fine-tune the student on those pairs. Frontier capability transfers at a tiny fraction of the training cost.

A concrete example

DeepSeek's R1 release shipped six distilled models: Qwen and Llama checkpoints from 1.5B to 70B fine-tuned on roughly 800k reasoning traces generated by the full R1. The distilled 7B beats far larger non-reasoning models on math benchmarks and runs on one consumer GPU.

Why it matters

Distillation is how frontier capability spreads to cheap, local, and open models - and why labs guard their APIs. Capability leaks through outputs: the NSA, FBI, and CISA just accused Chinese labs of doing it at industrial scale, and most labs' terms ban training competitors on outputs precisely because it works.