The Skew That Kills Quietly

Training computed avg_order_value_30d in pandas over a warehouse dump. Serving recomputes it in a service a backend engineer wrote from the spec doc. Two code paths, one feature name.

The differences are tiny, and they never throw errors — a timezone boundary, a different null-fill, a join that silently dropped rows. They just feed the model inputs it never trained on, every request, forever. Training-serving skew is the most common silent killer in production ML, precisely because it's invisible to every test that only checks 'did it return a number'.