BERT & the Pretrain–Fine-Tune Era (2018)

BERT (Devlin et al., 2018) took the transformer's encoder half and trained it on a deceptively simple task: mask random words and predict them (plus guess whether two sentences follow each other). No task labels needed — the raw text supervises itself.

Then the killer move: fine-tune that one pretrained model on eleven different NLP tasks — question answering, sentiment, entailment — and set records on nearly all of them by adding just a small output layer.

What it showed: a single massive self-supervised pretraining run could serve as a general-purpose language backbone. You no longer trained a model per task — you adapted one.