Spec'ing the Probabilistic
Shipping AI Features Safely
7 min+70 XPStep 1/11
Ship behind flags and gates
The AI shipping checklist differs from classic features: a feature flag (instant kill without deploy), an eval gate (the suite must pass the bar before the flag opens), staged rollout — internal dogfood → alpha cohort → expansion — with acceptance metrics watched per stage, and a rollback story that includes the model version. 'Revert' means reverting prompt + model + retrieval config, not just code.
If you can't answer 'which model version served that output' from your logs, you're not ready to ship.