Convolutional Networks and Computer Vision
A fully connected layer on a photograph would need 150 million weights and would have to learn every object once for every position. This unit replaces it with a small kernel that slides over the image, computes a convolution by hand, shows how the kernels are learned by backpropagation, and follows the design from LeNet-5 in 1998 to AlexNet in 2012 and the residual networks after it. It ends with transfer learning, which is how a few hundred medical images become a working classifier, and with adversarial examples, which show that the network does not see as we do.
A colour photograph of 224 by 224 pixels is a list of 150,528 numbers, and the networks of Units 6 and 7 would connect every one of them to every neuron of the first layer. This unit shows why that design fails for images and what replaced it: a small grid of weights, the kernel, that slides across the picture and treats every neighbourhood with the same weights. Networks built from such kernels read handwritten digits and bank cheques in the 1990s, won the ImageNet challenge of 2012 by a margin that changed the field, and today read retinal photographs and skin lesions. The unit ends with what these networks do not do. They do not see as we do, and a change too small for a person to notice can turn a panda into a gibbon.
1 Images and fully connected layers
A grey-scale image is a grid of pixels, each a number between 0 (black) and 255 (white), and Unit 1 turned the 28 by 28 grid of an MNIST digit into a vector of 784 components by reading the rows one after another. A colour image has three such grids, one each for red, green, and blue, called its channels, so a photograph of 224 by 224 pixels, the standard input size of the ImageNet networks of Section 5, is \(224 \times 224 \times 3 = 150{,}528\) numbers. A fully connected layer of the kind Unit 6 built gives every neuron one weight per input number, and the first worked example counts what that costs.
The parameter count is the first of three problems. Unit 4 showed what happens when a model has far more parameters than the data can constrain, and 1.2 million training images are a thin constraint on 150 million weights in one layer. The second problem is that a fully connected layer contains no notion of a neighbourhood. Its weights are a table with one entry per input component, and nothing in the table records that pixel 215 sits next to pixel 216 and above pixel 439. If every image in the training and test sets were scrambled by the same fixed shuffle of the pixels, a fully connected network would learn exactly as well as before, because the shuffle only renumbers its inputs; for a photograph that is absurd, because the pixels that belong together are the ones that lie together. The third problem is position. A cat in the top-left corner of a picture excites one set of weights and the same cat in the bottom-right corner excites a different set, so a fully connected network must learn what a cat looks like once for every place it can appear.
1.1 The transpose
Before the remedy comes a short piece of notation, needed in the worked example of Section 2 and in every later unit. Unit 1 wrote a vector as a column of numbers and the dot product as \(\mathbf{w}\cdot\mathbf{x}\). In the matrix language of the field the same product is written with the transpose.
For \(\mathbf{w}^{\top}\mathbf{x}\) the shapes are 1 by \(d\) times \(d\) by 1, which gives 1 by 1, a number, by the rule for matrix products of Unit 1. The backward pass of Unit 6 already used the transposed weight matrix \(\mathbf{W}^{\top}\) to send the error signal back through a layer; the forward product \(\mathbf{W}\mathbf{x}\) and the backward product \(\mathbf{W}^{\top}\boldsymbol{\delta}\) use the same table of weights in the two directions. From here on the dot product is written \(\mathbf{w}^{\top}\mathbf{x}\) wherever matrices are near, and \(\mathbf{w}\cdot\mathbf{x}\) where it reads better.