Crabhaus

What it is

Using a strong LLM to grade another model's outputs - scoring answers against a written rubric, or picking the better of two responses - in place of human raters or exact-match metrics. It now powers most free-text evals, leaderboards, and automated regression suites.

How it works

The judge gets the task, the candidate output, and grading instructions, and returns a structured verdict. Pairwise mode picks between two answers; single-answer mode scores against a rubric; reference-guided mode adds a known-good answer. Careful pipelines swap answer order, force a fixed verdict format, and first check the judge against human-labelled examples.

A concrete example

Testing a cheaper model for a support agent: a judge compares old vs new drafts on 100 real tickets, run twice with positions swapped - verdicts that flip with the ordering are discarded as noise. The judge is trusted only after it agrees with 30 human-labelled pairs about as often as a second human would.

Why it matters

Human evaluation is too slow and costly for every prompt tweak, and string matching can't grade prose. The MT-Bench work measured GPT-4 agreeing with human preferences over 80% of the time - human-human level - making evals cheap enough for CI. But judge biases flow silently into leaderboards, training rewards, and ship decisions.