Gradient Descent & Optimizers

These are my notes from Qiang Liu’s Machine Learning II course at UT Austin, cleaned up and stitched into a single story (with much help from Claude).

Almost everything in machine learning eventually comes down to the same move: you have a loss function that measures how wrong your model is, and you want to make it smaller. The model has parameters $\theta$ — sometimes a handful, sometimes a few hundred billion — and somewhere in that enormous space is a setting that makes the loss small. The whole game is finding it.