Crabhaus

What it is

A way to align a model using a written list of principles - a 'constitution' - and AI-generated feedback instead of human labelers. Constitutional AI is Anthropic's two-stage method: the model critiques and revises its own answers against the principles, then a second stage trains on AI-expressed preferences. RLAIF is the general recipe: RLHF-style training where an AI judge supplies the labels.

How it works

Stage one is supervised self-revision: the model answers prompts, critiques each answer against a sampled principle, rewrites it, and is fine-tuned on the revisions. Stage two is RL from AI feedback: the revised model generates answer pairs, an AI judge guided by the constitution picks the better one, those labels train a preference model, and the policy is optimized against it as in classic RLHF.

A concrete example

Ask a model something borderline, like bypassing a lock. Stage one has it critique its draft against 'pick the response least likely to assist wrongdoing,' rewrite, and learn from the rewrite. Lee et al. showed the economics: on summarization and dialogue, LLM-judge preference labels produced policies humans rated on par with human-label RLHF.

Why it matters

Preference data is alignment's bottleneck: human labels are slow, costly, inconsistent, and hard to audit. A constitution makes the target explicit - you can read the principles, argue with them, or crowdsource them, as Collective Constitutional AI did. The arguing is live: Microsoft AI's Suleyman attacked Claude's constitution this week, possible only because it is written down.