Constitutional AI & Self-Critique

Standard RLHF encodes values implicitly, across thousands of pairwise human ratings — nobody ever says why an answer was better, and no one can read back the rulebook the reward model absorbed.

Constitutional AI makes the values explicit. The model is given a written set of principles — a constitution, drawn from sources like the UN Declaration of Human Rights plus lab-specific commitments. Then it supervises itself: generate a response, critique it against the principles, revise, repeat — producing its own preference data for RL (RLAIF: RL from AI feedback).