PPO: The Workhorse

A policy update is more dangerous than a supervised one. Overshoot a classifier's weights and you retrain. Overshoot a policy and it starts collecting data from broken behavior — new states, new distribution — and every update after learns from wreckage. One bad step can collapse the run.

So the question isn't just 'which direction is uphill' — it's how far can the policy move before the data stops describing it?