The ML Lifecycle
Models Rot on the Shelf
The day you ship is the day it starts dying
A normal service fails loudly — an exception, a crash loop, a wall of 500s. A model fails quietly. The world moves, the weights don't, and every request still comes back in 12ms with a confident number attached.
That number drifts away from reality from the moment you deploy. User behavior shifts. Upstream systems change. Attackers adapt specifically to your model. The industry calls the family of phenomena drift, and it's not an edge case — it's the default state of every model in production.
Google's 'hidden technical debt' paper put it bluntly: the model is a tiny fraction of the system. The rest of the system is what keeps it honest.
insight
The most dangerous model failure isn't a crash — it's a slow fade every dashboard calls green, because 'returned a number' is not the same as 'was right'.