FLOPs, Bandwidth & the Roofline

A FLOP is one floating-point operation — a multiply, an add. An m×n matrix times an n×k matrix costs roughly 2mnk FLOPs. For transformer training the field uses a beautiful rule of thumb:

C ≈ 6 · N · D — six FLOPs per parameter per token, where N is parameters and D is tokens (2 for the forward pass, 4 for backward).

Run it: a 7B model on 1T tokens needs 6 × 7e9 × 1e12 ≈ 4.2e22 FLOPs. One H100 delivers ~990 TFLOP/s peak, but realistic utilization is ~40–50%, so call it 4e14 effective. That's 1e8 seconds — roughly 1,200 GPU-days, or about 60 GPUs running flat-out for three weeks.