Crabhaus

What it is

Judging an AI agent by the full path it took - every tool call, intermediate decision, and recovery - and by the final state of the world it changed, rather than only grading the text of its final answer.

How it works

Each run is logged as a trajectory: tool calls, arguments, results, retries, cost. Outcome scoring checks the end state - tau-bench diffs the database against an annotated goal, so a polite reply with the wrong booking still fails. Trajectory scoring checks the path: right tools, right order, no loops. Because agents are nondeterministic, pass^k demands k successes in a row on the same task.

A concrete example

An agent is asked to move a customer to an earlier flight. Answer-only eval sees 'Done! You're booked on the 09:15' and passes. Trajectory eval replays the run and finds modify_booking used last week's reservation ID - or was never called. In tau-bench retail, top agents passed under half of tasks once, and fell below 25% at eight-in-a-row.

Why it matters

Production agents fail through side effects: wrong records mutated, emails sent twice, silent loops burning tokens. None of that shows in the final message, so answer-grading overestimates reliability - and step-level scores show which stage broke when you swap a model, prompt, or tool.