Crabhaus

What it is

The post-training stage that turns human judgments - 'response A is better than B' - into a training signal. RLHF does it indirectly: fit a reward model on comparisons, then optimize the policy against it with reinforcement learning. DPO collapses that pipeline into one supervised-style loss applied directly to the same preference pairs.

How it works

Classic RLHF runs PPO against a learned reward model, with a KL leash keeping the policy near a reference model - four networks in play. DPO's insight: that KL-constrained objective has a closed-form optimum, so the reward can be rewritten in terms of the policy itself. What remains: a logistic loss raising the log-prob margin of chosen over rejected answers vs the frozen reference.

A concrete example

Zephyr-7B was tuned with distilled DPO on UltraFeedback comparison data and outperformed the far larger Llama 2 Chat 70B on MT-Bench. Meta's Llama 3 post-training likewise ran repeated rounds of SFT plus DPO on preference pairs rather than a PPO pipeline.

Why it matters

Preference tuning is where an assistant's tone, refusals, and judgment get shaped, and it was the most fragile, expensive part of post-training. DPO turned it into a single fine-tuning run on static pairs - cheap enough for small teams, now the default in open-weight stacks. It explains today's split: offline DPO for most products, online RL where labs need fresh samples, as in reasoning training.