GPT-2 → GPT-3 & the Scaling Laws (2019–2022)

GPT-2 (2019) made headlines partly for what OpenAI initially wouldn't release. The claimed surprise: a 1.5B-parameter model trained only to predict the next token could perform tasks it was never trained on — answer questions, summarize, translate — just by being conditioned on examples in its input.

What it showed: zero-shot task transfer was real. The next-token objective, over enough data, implicitly learns tasks. The paper's framing said it outright — 'Language Models are Unsupervised Multitask Learners.'