Crabhaus

What it is

A router is a small model or classifier in front of several LLMs that decides, per request and before any generation happens, which model should answer. Easy prompts go to a cheap, fast model; hard ones to a frontier model - so the strong model's price is paid only on traffic that needs it.

How it works

The router scores each prompt's difficulty and commits in one dispatch. Simple routers use heuristics like length or presence of code; learned routers such as LMSYS's RouteLLM train on preference votes between a strong and weak model's answers, predicting when the cheap one would satisfy. A threshold dial trades cost against quality; RouterBench measures routers on exactly that curve.

A concrete example

A coding assistant sends 'rename this variable' to a small open model for a fraction of a cent, and 'find the race condition in this async queue' to a frontier model. RouteLLM reported over 2x cost reduction on benchmarks with no meaningful quality loss - and the router kept working when the model pair behind it was swapped.

Why it matters

At production volume most requests are easy, and defaulting to the strongest model means frontier prices for autocomplete-grade work. Routing fixes that at the front door without touching any model - and it is quietly becoming infrastructure, as gateways and agent harnesses pick a model per step rather than pinning one.