Crabhaus

What it is

Structured adversarial testing: deliberately attacking your own AI system to find ways it can be broken before real adversaries do. The name comes from security practice, where a 'red team' plays the attacker. The target can be the model itself - can it be talked into harmful output? - or the application around it - can hostile data hijack an agent's tools?

How it works

Start from a threat model. Jailbreaks attack the model's own safety training with personas and encoding tricks; prompt injection attacks the application by hiding instructions in content it reads. Manual experts probe creatively; automated attacks scale coverage: attacker LLMs mutate prompts, and search methods like GCG find adversarial suffixes that transfer across models.

A concrete example

Before shipping an email agent that can file expenses, testers hide instructions in calendar invites and PDF attachments to see whether it obeys a stranger's text, while an automated harness fires thousands of known jailbreak patterns at the underlying model. Every successful attack becomes a permanent regression test the next model version must pass.

Why it matters

Once a model can browse, run code, and touch email or money, a successful attack does things instead of saying things. Red-teaming maps your real attack surface before attackers do it for you; frontier-lab safety frameworks and emerging regulation now expect it documented; and it only pays off when findings loop back into training data, guardrails, and regression suites.