Overfitting Deep Nets

A network with ten million parameters and ten thousand examples doesn't have to learn the pattern — it can memorize the examples. Given enough epochs it drives training loss to zero while test performance quietly degrades.

Deep learning's strange twist: modern nets are deliberately over-parameterized and still generalize — but only because a stack of regularization tricks keeps memorization in check.