Crabhaus

What it is

A design choice for feeding knowledge to an LLM: stuff everything into one giant prompt (long context) or retrieve a few relevant chunks first (RAG). 'Lost in the middle' names the measured failure where models read the edges of a prompt far better than its middle.

How it works

Stanford researchers moved the answer-bearing document around inside long prompts and plotted accuracy by position: a U-curve, strong at the start and end, sagging in the middle, even for long-context models. NVIDIA's RULER shows most models degrade well before their advertised limit as tasks harden. RAG shrinks the prompt to retrieved passages, at the price of missing what retrieval misses.

A concrete example

A Google DeepMind team benchmarked both head to head: with a strong model and enough budget, full long-context prompting edged out RAG on accuracy, but RAG was dramatically cheaper. Their Self-Route method lets the model route each query, sending easy ones through RAG and hard ones through the full window, keeping near-long-context accuracy at a fraction of the token cost.

Why it matters

Million-token windows make 'just paste everything in' tempting, but tokens cost money and latency, and mid-prompt facts still get skipped. For agent builders the practical pattern is hybrid: retrieve first, keep prompts tight, and place load-bearing instructions or evidence near the start or end of the window.