Crabhaus

What it is

A repeatable test suite for an AI feature: a fixed set of real inputs with known-good expected outputs - the golden set - plus a scorer that grades the model's answers. You run it every time you change a prompt, swap a model, or touch the pipeline, exactly like unit tests for ordinary code.

How it works

Collect 30-100 real examples of your task, label each with the expected answer or a pass/fail rubric. A runner replays every input through the current prompt and model, then scores outputs: exact-match or assertions for structured tasks, an LLM judge with a written rubric for free text. A change ships only if the score holds or improves; the judge itself is spot-checked against human labels.

A concrete example

A support-bot team keeps 60 real tickets labeled with the correct category and refund decision. When a cheaper model launches, they rerun the suite: it matches 58 of 60 - but flips two refund-eligible tickets to deny. The models felt identical in casual use; the golden set surfaced a costly regression in minutes.

Why it matters

LLM behaviour shifts in non-obvious ways with every prompt edit or model update. Without a fixed test set, teams judge changes by re-trying favourite prompts and quietly break edge cases. A golden set turns 'feels better' into a measured delta, makes model swaps a ten-minute decision, and is cheap: a few dozen well-chosen examples catch most regressions.