Training LLMs
The Next-Token Game
8 min+80 XPStep 1/10
One game, played trillions of times
An LLM's entire pretraining objective: predict the next token. Given 'The capital of France is', predict 'Paris'. Show it essentially the entire internet, billions of times.
To predict well, the model must learn grammar, facts, reasoning patterns, code — everything that helps predict what comes next. Scale turned this simple game into general capability.
analogy
Like a musician who's heard every song ever made trying to guess the next note — eventually they internalize music theory itself.