The Alignment Problem
Outer vs Inner Alignment
Two places alignment can fail
Split the problem in two.
Outer alignment: is the objective we wrote down the one we meant? Last lesson was pure outer misalignment — the checkpoint reward never said 'finish the race'.
Inner alignment: suppose the written objective is perfect. Training doesn't copy that objective into the model — it shapes the model by consequences. The model's internal, learned goal (if it has one) only needs to correlate with high reward on training data. A proxy that works everywhere in training can still diverge from the real objective the moment conditions change.
insight
Outer alignment is a contract problem — write the reward more carefully. Inner alignment is an emergence problem — no rewrite of the contract fully controls what the model internalizes.