Image: Derrick Coetzee, via Wikimedia Commons, CC0
Unit 6 gave the gradient; this unit gives the practice that turns it into a trained network. It explains why the gradient is estimated on a minibatch, how to read a learning rate off a loss curve, what momentum and Adam add and why Adam's learning rate is so small, how the gradient vanishes through twenty sigmoid layers and what the ReLU function, batch normalization, and residual connections did about it, how weight decay, dropout, early stopping, and augmentation keep a network honest, and what a training run costs in FLOP, GPU-hours, and dollars.
Preview: the beginning of the unit
Unit 6 ended with an algorithm that computes the gradient of the loss with respect to every weight of a network, and with the gradient descent update of Unit 4 that uses it. Between that algorithm and a network that reads photographs lie a few dozen decisions, each small and each able to ruin the run: how many examples to look at before a step, how large a step to take, where to start, how to keep the gradient from fading through many layers, how to keep a network with a hundred thousand weights from memorizing its training set, and what the whole thing costs on a graphics processor. This unit is about those decisions, most of which were settled after 2012.
1 From the gradient to the step
The training loss of Unit 4 is an average over the \(N\) training examples,
\[
L(\boldsymbol{\theta}) = \frac{1}{N}\sum_{i=1}^{N} \ell_i(\boldsymbol{\theta}) ,
\]
where \(\ell_i\) is the loss of example \(i\) and \(\boldsymbol{\theta}\) holds all weights and biases. Its gradient is the average of the gradients of the \(\ell_i\), so one exact gradient step needs one forward pass and one backward pass for every example. On the 60,000 images of MNIST that is 60,000 backward passes before a single weight moves, and a few thousand such steps are needed.
The cheaper method is to estimate the gradient from a sample. A minibatch is a small set of \(B\) training examples drawn at random, and the average of their gradients is an estimate of the full gradient. Each example is drawn from the training set with the same probability, so the estimate has the right value on average, and it is wrong by a random amount that shrinks with the square root of \(B\). This is the same rule of probability that makes the average of \(B\) noisy measurements \(\sqrt{B}\) times less noisy than a single one. Taking a step along such an estimate is called stochastic gradient descent. The idea goes back to a 1951 paper by Herbert Robbins (1915-2001) and Sutton Monro (1919-1995) on finding the root of a function that can only be measured with noise, and nearly every network since the 1980s has been trained this way. One pass through the whole training set, in minibatches, is called an epoch; the training set is shuffled before each epoch so that no two epochs present the same batches.