When Offline Metrics Lie

Offline metrics — AUC, nDCG, RMSE, precision@k — are computed on logged interactions, and the logs only contain items the previous policy chose to display. So they answer 'did the model rank the shown items correctly' — not 'would the feed be better'.

The candidates your new model would surface have no logged labels at all. This is why offline evals are necessary for CI (catch regressions cheaply) and insufficient for ship decisions (they can't see the counterfactual). The divergence between the two is where recommendation science gets interesting — and where naive eval pipelines get misleading.