Monitoring & Maintenance
Incidents, Rollbacks, and the Retro Loop
10 min+110 XPStep 1/10
A model incident is still an incident
Same SRE discipline, stranger failure shapes. Severity is measured by blast radius — who's affected, whether it's silent or loud, whether money or safety is on the line.
The playbook doesn't change: detect (monitoring, hopefully — not a customer's tweet) → mitigate → diagnose → fix → retro. Mitigate always means the same first move: the kill switch — a config flag that routes traffic back to the last-known-good model, or to a dumb-but-honest heuristic. Investigate later; stop the bleeding now.
warning
If your rollback plan is 'retrain from the last checkpoint', you don't have a rollback plan — you have a six-hour outage and a fondness for notebooks.