What a convolutional network sees Image: Alvesgaspar (CC BY-SA 3.0)

What a convolutional network sees

A photograph of 224 by 224 pixels in colour is 150,528 numbers. A fully connected layer of 1,000 neurons on top of it has 150 million weights, one from every pixel to every neuron, and nothing in that layer knows that two pixels are next to each other. A convolutional layer with 64 kernels of 11 by 11 on the same image has 23,296 weights and knows nothing else. That difference, and the fact that it works, is the subject of this article.

Nine products and a sum

Take a 5 by 5 grey-scale image in which the left three columns are 0 and the right two columns are 10: a vertical edge. Take a 3 by 3 kernel

\[ \mathbf{K} = \begin{pmatrix} -1 & 0 & 1 \\ -1 & 0 & 1 \\ -1 & 0 & 1 \end{pmatrix}. \]

Place the kernel over the top left 3 by 3 patch of the image, multiply each of the nine kernel numbers by the image number underneath it, and add. In that patch every image value is 0, so the sum is 0. Slide the kernel one column to the right. Now its right column sits over the edge, where the image values are 10, so the sum is \(10 + 10 + 10 = 30\). Slide once more: the right column is over values of 10 and the left column over 0, sum 30 again. Move down a row and repeat. The output is a 3 by 3 feature map

\[ \begin{pmatrix} 0 & 30 & 30 \\ 0 & 30 & 30 \\ 0 & 30 & 30 \end{pmatrix}. \]

The kernel fired wherever its window straddled the edge and stayed quiet where the image was flat. It is a vertical edge detector, and it would respond nowhere on an image with a horizontal edge. That is the whole operation: a small table of weights, slid across the image, with a dot product at every position. The output is smaller than the input by the kernel size minus one in each direction (5 - 3 + 1 = 3); padding the image with a border of zeros keeps it the same size, and a stride of 2 moves the kernel two pixels at a time and halves it.

Why the weights are shared

The nine numbers of the kernel are the same at every position. That is where the parameter count comes from, and it is also an assumption about images: a feature worth detecting in the top left is worth detecting in the bottom right. An edge is an edge wherever it is, and so is a cat's ear. The mathematical name for the property is translation equivariance: shift the image and the feature map shifts with it. Pooling, which replaces each 2 by 2 block of a feature map by its maximum, throws away the exact position and makes the network's answer the same whether the cat is a few pixels left or right.

The counts for a real first layer: 64 kernels, each 11 by 11 and reaching across the 3 colour channels, is \(64 \times 11 \times 11 \times 3 = 23{,}232\) weights plus 64 biases. The fully connected alternative was 150 million. Fewer parameters means less data needed to fit them and less overfitting, and the convolution runs as a matrix multiplication that a graphics card performs very fast.

The move that made it learn

The edge kernel above was written by hand, and image processing used hand-written kernels for decades. The move that defines the convolutional network, made by Yann LeCun and colleagues from 1989 and described in full in their 1998 paper on reading handwritten digits and cheques, is to treat the nine numbers as weights to be learned by backpropagation, like any other weights. The network is shown digits and told the right answers, and the kernels of the first layer settle into whatever detectors reduce the error. Nobody tells it to look for edges.

When researchers look at what the first layer learned, on digits or on photographs, they find edges at every orientation, bars, and blobs of colour, in pairs of opposite contrast. The 2012 AlexNet paper shows its 96 first-layer kernels as a grid of small images, and they look like the oriented edge and colour detectors that David Hubel and Torsten Wiesel found in the visual cortex of cats in 1959. Deeper layers combine them: a corner from two edges, a texture from many bars, an eye from a dark blob ringed by a light one, and by the fifth layer there are feature maps that respond to faces and wheels. None of this hierarchy is programmed. It is what falls out of gradient descent on labelled images, which is the single most important empirical fact about computer vision.

From LeNet to the present

LeNet-5 of 1998 had two convolutional layers, two pooling layers, and three fully connected ones, about 60,000 parameters, and read handwritten digits with an error of under 1 per cent; it processed a large share of the cheques deposited in the United States in the late 1990s. AlexNet of 2012 had five convolutional layers and about 60 million parameters, and won the ImageNet challenge by a margin that moved the whole field, as the article on AlexNet tells. The networks of 2015 had 152 layers, made trainable by a shortcut around every pair of layers, and matched or passed the human error rate on the same benchmark.

Most people who use these networks today never train one from scratch. They take a network trained on ImageNet, keep its learned kernels, and retrain only the last layer on a few hundred images of their own problem, such as X-rays or satellite pictures. That is transfer learning, and it works because the edges, textures, and parts that the early layers learned on photographs of everything are useful for photographs of anything.

What it does not see

A network that gets 95 per cent of photographs right does not see as we do. Adding a pattern of tiny changes to a photograph, invisible to a person, can make the network call a panda a gibbon with high confidence; the changes are computed by the same gradient that trained the network, pointed at the input instead of the weights. Such adversarial examples show that the network's features, however humanlike the first layer looks, are not the features we use, and the fix is an open research problem. It is also a reminder to read "95 per cent accuracy" as a statement about a test set and nothing more.

Unit 9 of the AI Masterclass computes this convolution in full, works out the output sizes and parameter counts of LeNet-5 layer by layer, and explains receptive fields, transfer learning, and adversarial examples. The tool of that week, CNN Explainer, shows a real network doing all of it on a photograph you upload.

Learn how machines learn, properly

The AI Masterclass: weekly units, quizzes, a final exam, and a certificate.
About the program