Gradient Descent, Properly

Nearly every model you've heard of — GPT, image generators, recommenders — was trained by some flavor of gradient descent. The gradient points uphill; we step downhill; repeat.

The learning rate controls step size. It's the single most important hyperparameter: too large, training diverges; too small, it never finishes.