The Hardware Lottery

Backpropagation was published in 1986. For the next twenty-five years, neural networks mostly lost — to SVMs, to boosted trees, to hand-engineered features.

What changed wasn't the math. Researcher Sara Hooker calls it the hardware lottery: ideas don't win on elegance — they win when they happen to fit the silicon available. Neural nets are billions of small, independent multiply-adds — a terrible fit for a CPU that does one thing fast, and a perfect fit for a GPU.