Crabhaus

What it is

A safety classifier is a separate model that scores content against policy harm categories - violence, self-harm, harassment - instead of replying. Moderation systems run one pass on what users send in and another on what the model produces; the scores feed whatever enforcement the application chooses. The classifier supplies the signal, the product decides what happens.

How it works

The classifier maps text or images to a 0-1 score plus a flag per category. OpenAI's omni-moderation endpoint screens both sides and can return scores alongside a generation. Llama Guard fine-tunes an LLM to judge against a taxonomy written in its prompt, so policies swap without retraining. Anthropic's constitutional classifiers train guards on synthetic data from a written constitution.

A concrete example

A support bot calls OpenAI's free moderation endpoint on each message. One scores 0.91 on harassment, over the app's 0.8 threshold, and is blocked before the main model sees it; a 0.45 self-harm score falls below the block line but routes to a human reviewer with a helpline template. The same code screens outputs before display.

Why it matters

Agents accept arbitrary input and produce unsupervised output; classifiers are the cheap screening layer that makes that tolerable. Thresholds are the real product decision: too low and false positives block real users, too high and harms slip through. Coverage gaps bite - omni-moderation handles text and images but not audio, so an untranscribed voice agent is unscreened.