Crabhaus

What it is

A cost-control pattern where every request goes to a small, cheap model first and escalates to a bigger, costlier model only when the cheap answer looks unreliable. Instead of one model for all traffic, models are chained cheapest-to-strongest and a confidence check decides how far each query climbs. Formalized for LLM APIs in the 2023 FrugalGPT paper.

How it works

The small model answers; a scorer judges the answer via a trained verifier, token probabilities, or agreement across samples. Clear the threshold and the answer ships - the big model never runs. Fail it and the query re-runs on the next tier up. The threshold is the knob: FrugalGPT reported matching GPT-4 accuracy on several benchmarks at up to 98% lower cost, because most traffic is easy.

A concrete example

A support bot: a small local model drafts a reply to every ticket and a verifier scores it against the question. Password resets and order-status queries pass, costing fractions of a cent. A gnarly refund dispute fails the check and escalates to a frontier model, which bills only for tickets that need it. Easy tickets also come back faster.

Why it matters

Inference bills scale with the strongest model you make the default, and high-volume traffic rarely needs frontier quality on every call. A cascade converts that into direct savings without retraining anything, and it degrades gracefully: if the big model is down or rate-limited, the cheap tier keeps answering the easy majority.