Crabhaus

What it is

A hallucination is a confident, fluent output that is factually wrong: an invented citation, date, API, or event. It is not a malfunction layered on top of normal operation - it is ordinary next-token prediction producing plausible text that happens to be false.

How it works

Pretraining rewards plausibility, not truth: rarely-seen facts can't be separated from plausible inventions, so even a calibrated predictor errs on them. Evals lock the habit in - most grade pure accuracy, so guessing outscores admitting uncertainty. Anthropic adds a circuit view: a 'known entity' feature can suppress Claude's default refusal; when it misfires, the model confabulates.

A concrete example

In Mata v. Avianca (2023), New York lawyers filed a brief with six case citations ChatGPT had invented, down to docket numbers and quotes. Their check was asking the model whether the cases were real; it said yes. A federal judge sanctioned them - the canonical proof that fluency and truth are independent properties.

Why it matters

Hallucination is a design constraint, not a bug the next release removes. It is why RAG grounds answers in retrieved text, why agents verify with tools instead of trusting recall, and why newer evals reward calibrated abstention. The mechanism tells you what helps - grounding, citations, uncertainty-aware training - and what doesn't: scale alone, or telling the model not to make things up.