InstructGPT & Chain-of-Thought (2022)

Raw LLMs predict text; they don't follow instructions. InstructGPT (Ouyang et al., 2022) fixed this with RLHF — reinforcement learning from human feedback — in three steps:

• Collect human-written demonstrations → supervised fine-tuning
• Have humans rank multiple model outputs → train a reward model on those preferences
• Optimize the model against the reward model (PPO), pushing outputs toward what humans prefer

The shocker: the resulting 1.3B model was preferred by raters to the 175B GPT-3. Alignment, not scale, was the missing piece — and 'helpful assistant' became a trainable property rather than a wish.