The Memory Wall

Two curves diverged for decades: compute throughput (FLOPs) grew fast — orders of magnitude per hardware generation — while memory bandwidth grew slowly. The result is the memory wall: for most workloads, the processor can compute far faster than it can be fed data.

The governing metric is arithmetic intensity: FLOPs per byte moved. Dense matrix operations at high arithmetic intensity use the processor well; memory-bound operations — where you read more than you compute — leave it starved. Whether a workload is compute- or memory-bound is decided by this ratio against the hardware's ridge point, not by intuition.