Red-Teaming & Capability Evals

Red-teaming borrows the security practice: dedicated humans (and increasingly, other models) try to break the system before it's released. They hunt jailbreaks — prompt structures that bypass refusals — and probe for the failure modes nobody designed for: leaked system prompts, instructions smuggled inside retrieved documents, outputs that are technically allowed and practically harmful.

Automated red-teaming scales the hunt: a model generates thousands of attack variants, keeps the ones that work, and the failures become training data. The loop is genuinely adversarial — every defense shipped teaches the next attack where to aim.