ML Forecasting & Evaluation
Never Leak the Future
8 min+80 XPStep 1/10
Random splits are time machines
Ordinary k-fold cross-validation shuffles rows and splits them — fine for independent data, disastrous for time series. Shuffle timestamps and your model trains on November to 'predict' March. It sees next season's patterns, next year's level, and every future shock — then reports gorgeous metrics for a skill nobody possesses: predicting the past from the future.
Temporal leakage is the most common silent killer of deployed forecasters. The model wasn't bad; the evaluation was a fantasy.
warning
If a backtest looks too good, ask one question first: could every feature have been computed using only data available at that timestamp? If the answer is no, the score is fiction.