Inside the Transformer
Scale Changes Everything
10 min+100 XPStep 1/10
Loss falls on a predictable curve
In 2020, researchers published the scaling laws: test loss falls as a smooth power law with model size, dataset size, and compute. Performance doesn't improve randomly — it follows a curve you can extrapolate.
That's how labs justify a single training run costing tens of millions of dollars: fit the curve on small models, and you can predict what a 100× bigger run will score before spending the money.
insight
The Chinchilla study (2022) corrected the recipe: most big models were *undertrained*. The efficient frontier is roughly 20 training tokens per parameter — a 70B model wants ~1.4 trillion tokens.