The Tricks That Made It Work

Consecutive RL experiences are correlated — frame 1001 looks like frame 1000. Training a network on that stream is like cramming one flashcard for hours: it overfits the recent past and forgets everything else.

Experience replay stores every transition in a buffer and trains on random samples drawn from it. That breaks the correlation, lets each experience teach the net many times — crucial when experiences are expensive — and smooths the distribution the network sees.