RLHF Under the Hood

RLHF is a three-stage machine. Stage one — SFT: fine-tune a base model on demonstrations of good assistant behavior, so 'helpful' is at least on the table. Stage two — reward model: humans rank pairs of model outputs; those comparisons train a separate network to predict which outputs humans would prefer. Stage three — RL: the assistant is optimized (historically with PPO) to score high under the reward model.

One detail does real work: a KL penalty tugs the policy back toward the SFT model at every step — a leash that keeps the optimizer from sprinting to wherever the reward model's quirks point.