Making Models Fit

Distributed Training

11 min+105 XPStep 1/10

When one GPU isn't enough — do the math

How much memory does training need? More than the weights suggest. In standard mixed-precision Adam, per parameter you store:

2 bytes — fp16 weight
2 bytes — fp16 gradient
4 bytes — fp32 master weight
8 bytes — Adam's two moments (momentum + variance)

That's ~16 bytes per parameter. A 7B model needs ~112 GB before a single activation — more than an 80 GB H100 holds. The 405B-scale runs need tens of thousands of GPUs, not because of FLOPs alone, but because the state doesn't fit anywhere smaller.