Image: Phillip Capper from Wellington, New Zealand (CC BY 2.0)What a gradient is and why every AI model follows one
A model with one parameter, a loss that is a bowl, and a learning rate of 0.25: start at \(w = 0\), and after three steps you are at 2.625, on the way to the minimum at 3. Change the learning rate to 1.1 and after the same three steps you are at 8.184 and moving away. That is the whole of gradient descent, and it is the same move that trains a language model with 175 billion parameters. The only difference is the number of parameters.
A slope is a number
The derivative of a function at a point is the slope of the curve there: how much the function rises for a small step to the right. For \(f(w) = (w - 3)^2\), the derivative is \(f'(w) = 2(w - 3)\). At \(w = 0\) the slope is \(-6\), which means the curve is falling steeply as \(w\) increases. At \(w = 5\) the slope is \(+4\), rising. At \(w = 3\) the slope is 0, and that is the bottom of the bowl.
The sign of the slope tells you which way is downhill. If the slope is negative, the function gets smaller when you move right, so move right. If it is positive, move left. In one line: move in the direction opposite to the sign of the derivative. The size of the slope tells you something too. Far from the bottom the slope is large and you can afford a big step; near the bottom it is small and the steps shrink by themselves.
With several parameters the derivative becomes a list of partial derivatives, one per parameter, called the gradient. For a function of two variables \(f(x, y)\) the gradient is \((\partial f/\partial x, \partial f/\partial y)\), and it points in the direction in which \(f\) increases fastest. The direction opposite to it is the steepest way down. A model with a billion parameters has a gradient with a billion components, and the rule is the same: subtract a multiple of the gradient from the parameters.
Three steps, with the numbers
The update rule is
\[ w \leftarrow w - \eta\, f'(w), \]where \(\eta\) (the Greek letter eta) is the learning rate, a small positive number that you choose. The arrow means "replace the old value by the new one". Take \(\eta = 0.25\) and start at \(w = 0\).
- Step 1. The slope at 0 is \(2(0 - 3) = -6\). New value: \(0 - 0.25 \times (-6) = 1.5\).
- Step 2. The slope at 1.5 is \(2(1.5 - 3) = -3\). New value: \(1.5 + 0.75 = 2.25\).
- Step 3. The slope at 2.25 is \(-1.5\). New value: \(2.25 + 0.375 = 2.625\).
Each step covers half the remaining distance to the minimum at 3. That is no accident: on this bowl the new distance is \((1 - 2\eta)\) times the old distance, and with \(\eta = 0.25\) the factor is 0.5. After ten steps the distance is \(3 \times 0.5^{10}\), about 0.003. The loss, which is the distance squared, is then about \(10^{-5}\).
The step that is too large
Now take \(\eta = 1.1\) from the same start.
- Step 1. Slope \(-6\). New value: \(0 + 6.6 = 6.6\). You have jumped over the minimum and landed farther from it than you started.
- Step 2. Slope at 6.6 is \(+7.2\). New value: \(6.6 - 7.92 = -1.32\).
- Step 3. Slope at \(-1.32\) is \(-8.64\). New value: \(-1.32 + 9.504 = 8.184\).
The factor is now \(1 - 2.2 = -1.2\): each step flips the sign and multiplies the distance by 1.2. The values swing from side to side with growing amplitude and the loss goes up without limit. This is what people mean when they say training "diverged". Nothing is broken in the algorithm; the step is simply too big for the curvature of the bowl. The border between the two behaviours on this function is \(\eta = 1\), where the factor is \(-1\) and the value bounces between 0 and 6 forever.
Real losses are not bowls of one variable, and nobody knows their curvature in advance, which is why the learning rate is the first thing a researcher tunes and the first thing to suspect when a run goes wrong. The article on learning rates shows what the four cases look like on a loss curve.
Where the loss comes from
In the example the function to minimize was given. In machine learning the function is built from data. A model has parameters, written \(\boldsymbol{\theta}\) for all of them together, and makes a prediction \(\hat{y}\) for each input. A loss measures how wrong the prediction is, such as \((y - \hat{y})^2\) for a number or the cross-entropy for a probability, and the loss over the training set is the average over all examples, \(L(\boldsymbol{\theta})\). The gradient \(\nabla_{\boldsymbol{\theta}} L\) says how the average error changes when each parameter is nudged, and the update is
\[ \boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \eta\, \nabla_{\boldsymbol{\theta}} L. \]For a straight line through data this gradient can be written out in two lines. For a neural network it is computed by the chain rule applied layer by layer, which is called backpropagation, and for a model with billions of parameters a library does it automatically. The update that follows is the same line in every case.
Two practical twists are worth knowing. Computing the gradient over millions of examples for each step is wasteful, so training uses a random batch of a few hundred examples per step and accepts a noisy estimate of the gradient; that is stochastic gradient descent. And the learning rate is usually not constant: it is often raised for the first few hundred steps and then lowered slowly, and the most used method, Adam, scales each parameter's step by the recent size of its gradient. Adam's usual starting learning rate is \(10^{-3}\), a number every practitioner carries in their head.
Why one idea runs the whole field
The reason gradient descent is everywhere is that it asks so little. It does not need a formula for the minimum, as least squares has. It does not need to try every combination, as a search would. It needs only that the loss be differentiable and that someone can compute the gradient, and with automatic differentiation that someone is a program. A loss of a different shape, a model of a different kind, a dataset of a different size: the training loop does not change.
It also promises less than textbooks sometimes suggest. On a bowl there is one minimum and gradient descent reaches it. On the loss surface of a network there are many valleys, flat regions, and saddle points, and the method finds a low point, not the lowest. That the low points it finds are good enough is one of the empirical facts of the field, and one that is still being explained.
In the AI Masterclass, Unit 1 introduces the derivative and the gradient and walks these three steps, Unit 4 applies the update to a real loss for the first time, and Unit 7 covers the practice: batches, learning rate schedules, momentum, and Adam.


