A/B Testing for Models

Every eval upstream of production is a proxy: offline suites measure the golden set's world, judges measure their own biases, human evals measure a sample. The A/B test measures the thing you actually care about — what happens when real users meet the real model.

This is why model decisions end in experiments: two variants, randomized traffic, measured on business outcomes. The eval suite gets you to the experiment; the experiment gets you to the decision. Skipping it means shipping a proxy's verdict as if it were the truth.