Q-Learning: What Is an Action Worth?

'How good is this state?' is the wrong question — you can't act on it. The actionable question is Q(s, a): the expected return if I take action a in state s, then play well afterward.

Know Q for every state-action pair and the policy is free: in any state, take the highest-Q action. Value-based RL bets it's easier to learn 'what is each move worth' than 'what should I do' directly.