Learning Rate & Schedules

Gradient descent needs a step size — the learning rate — and it's the single most consequential choice in training.

Too high: you leap over the minimum each step and the loss diverges or rattles forever. Too low: you creep downhill and run out of time or compute budget long before you arrive. There's no universally right value — it depends on the loss surface you're standing on.