Image: Ordercrazy (CC0)Why a learning rate that is too large ruins everything
Every person who has trained a network has watched a loss curve go the wrong way. The loss is the number the training loop is trying to make small, and the learning rate is the size of each step it takes. Set the rate ten times too small and the curve is a flat line that never reaches anything in the time you have. Set it ten times too large and the curve climbs, jumps to infinity, and the program prints "nan". Between the two there is a range that works, and the shape of the curve in the first few steps tells you which side of it you are on.
Four runs on the same bowl
The simplest loss is a bowl, \(L(w) = w^2\), with its minimum at 0. The gradient is \(2w\), and a step of gradient descent with learning rate \(\eta\) is \(w \leftarrow w - 2\eta w = (1 - 2\eta)\, w\). Start at \(w = 4\) and run five steps with four learning rates.
- \(\eta = 0.05\): 4, 3.6, 3.24, 2.92, 2.62, 2.36. The factor per step is 0.9. The loss, \(w^2\), goes 16, 13.0, 10.5, 8.5, 6.9, 5.6: falling, slowly, and after fifty steps \(w\) is still 0.02. Too small.
- \(\eta = 0.25\): 4, 2, 1, 0.5, 0.25, 0.125. Factor 0.5. The loss goes 16, 4, 1, 0.25, 0.06, 0.016. Each step quarters it. Right.
- \(\eta = 0.75\): 4, \(-2\), 1, \(-0.5\), 0.25, \(-0.125\). Factor \(-0.5\). The loss is the same as before, 16, 4, 1, 0.25, but \(w\) crosses the minimum at every step. Converging, by overshooting.
- \(\eta = 1.1\): 4, \(-4.8\), 5.76, \(-6.91\), 8.29, \(-9.95\). Factor \(-1.2\). The loss goes 16, 23, 33, 48, 69, 99. Diverging.
On this bowl the rule is exact: the run converges when \(|1 - 2\eta| < 1\), that is for \(\eta\) below 1, and the fastest rate is \(\eta = 0.5\), which reaches the minimum in one step. Real losses have more complicated shapes, and yet near a minimum almost every smooth function looks like a bowl, and the curvature of that local bowl sets the largest safe step. The trouble is that nobody knows the curvature in advance, it differs from parameter to parameter, and it changes during training.
Reading the curve
Those four runs are what the four kinds of loss curve look like. A curve that falls in a straight, shallow line and keeps falling at the end of training is the 0.05 case: raise the rate. A curve that drops steeply, then levels off smoothly as the steps shrink, is the 0.25 case. A curve that falls but wobbles, with the loss going up and down from step to step around a falling trend, is the 0.75 case: it is working, and a smaller rate would make it work more smoothly. A curve that rises, or that drops for a few steps and then jumps to a huge value, is the 1.1 case, and lowering the rate by a factor of 10 is the first thing to try.
There is a fifth curve to know, which is not about the learning rate. The training loss falls and keeps falling while the loss on held-out validation data falls, levels off, and then begins to rise. That is overfitting: the network has started to memorize the training examples, and the cure is to stop at the lowest point of the validation curve, which is called early stopping, or to regularize. A sixth is a curve stuck at exactly the loss of a model that always predicts the average, which usually means a bug: a learning rate of zero, a mislabelled dataset, or inputs that never reach the model.
Why the usual rate is so small
The bowl above suggests learning rates around 0.5. Real networks are trained with rates around \(10^{-3}\), and the reason is that the loss surface of a network with millions of parameters is a bowl that is very steep in some directions and very flat in others. The largest safe step is set by the steepest direction. A step that is safe there is tiny along the flat directions, and progress along them is slow. Momentum helps: it keeps a running average of past gradients, so that steps along a consistent direction build up speed while steps that flip sign cancel out. Adam, the method most networks are trained with, goes further and divides each parameter's step by the running size of its own gradient, so that steep and flat directions get steps of similar size. Its usual starting rate of \(10^{-3}\) is a rule of thumb that works for a surprising range of models, and the first thing a researcher does on a new problem is try \(10^{-2}\), \(10^{-3}\), and \(10^{-4}\) and look at the three curves.
The rate is also not held constant. Most training runs start with a warm-up, a few hundred steps during which the rate climbs from near zero to its target, because at the start the random weights make the loss surface a bad bowl and a full-size step would throw the parameters somewhere worse. After warm-up the rate is lowered slowly, in steps or along a cosine curve, so that the final steps are small enough to settle into a minimum rather than bounce around it. The schedule is reported in every paper's training details, and a run that cannot be reproduced usually differs from the paper in exactly this.
The batch and the noise
One more thing changes the picture. The gradient in a real run is computed over a random batch of, say, 128 examples rather than over the whole training set, which makes each gradient a noisy estimate of the true one. The noise adds a jitter to the loss curve that is not the oscillation of the 0.75 case, and a plotted curve is usually smoothed to see through it. Larger batches mean less noise and allow larger learning rates; the two are tuned together. The jitter is not entirely a nuisance. There is evidence that the noise helps the run find flat minima that generalize better, which is one reason small batches are still used when memory would allow larger ones.
Unit 7 of the AI Masterclass plots the four curves of this article on the bowl, works three steps of Adam by hand with its bias correction visible, and explains initialization, batch normalization, dropout, and early stopping. The article on the gradient covers the step itself.


