RLHF: RL Meets Language Models

A pretrained LLM learned one thing: predict the next token. Hand it 'How do I fix a flat tire?' and the most likely continuation might be a forum quote, another question, or a poem — the internet contains all three. Helpful, honest, harmless assistant behavior exists in the distribution, but raw prediction doesn't select for it.

And 'helpfulness' can't be written as a clean reward function — nobody can formula-score it. But humans are excellent at one cheap thing: look at two answers and say this one's better.