AlexNet and the 2012 result that changed artificial intelligence Image: Fritzchens Fritz from Berlin (CC0)

AlexNet and the 2012 result that changed artificial intelligence

On 30 September 2012 the results of that year's ImageNet Large Scale Visual Recognition Challenge were released. The task was to label photographs with one of 1,000 categories, and an entry was scored on its top-5 error: the share of test images for which the correct label was not among its five guesses. The second-placed entry, from a team using the best hand-designed features of the time, had an error of 26.2 per cent. The winner, a neural network from the University of Toronto called AlexNet, had 15.3 per cent. In a competition where a point or two had separated the leaders every year, the gap was eleven points.

What the network was

AlexNet was built by Alex Krizhevsky with Ilya Sutskever, supervised by Geoffrey Hinton. It had five convolutional layers and three fully connected ones, about 60 million parameters, and 650,000 neurons. The convolutional layers were the kind the article on convolutional networks describes, learned by backpropagation as LeCun had done since 1989. The architecture was not new in kind. It was much larger than anything that had been trained on photographs before, and three choices made that size trainable.

The first was the activation function. Instead of the sigmoid or the tanh that networks had used for decades, AlexNet used the rectified linear function, \(\max(0, z)\), which does not flatten out for large inputs and so does not strangle the gradient on its way back through many layers. The paper reports that it trained several times faster. The second was dropout: during training, each neuron in the fully connected layers was switched off at random with probability 0.5, so that no neuron could rely on any particular other one, which cut the overfitting that 60 million parameters invite. The third was data augmentation, feeding the network shifted, flipped, and colour-perturbed copies of the training images so that it saw many more pictures than there were.

Six days on two graphics cards

The network was trained on two NVIDIA GTX 580 graphics cards, each with 3 gigabytes of memory, consumer products sold for video games. Neither card could hold the whole network, so it was split across the two, with the layers communicating only at certain points, which is why the published architecture diagram has two parallel streams. Training took between five and six days. Krizhevsky wrote the GPU code himself, and the speed of a graphics card at the matrix multiplications of a convolution, thousands of them at once, was the difference between an experiment that finished in a week and one that would have taken months on the processors of the time.

That is the compute half of the story. The data half had been built by Fei-Fei Li's group, first at Princeton and then at Stanford, from 2007: ImageNet, a collection of 14 million photographs labelled by crowd workers according to the categories of a dictionary of nouns, of which the challenge used 1.2 million training images in 1,000 categories. Nothing of that size had existed. The methods half, backpropagation and convolutional layers, had waited since 1986 and 1989 for data and compute to catch up. The usual summary of 2012 is that the three came together, and it is right.

What the numbers meant

A top-5 error of 15.3 per cent means that on about one image in seven, the right category was not among the network's five guesses. The categories were fine: 120 breeds of dog among the 1,000, and the difference between a Siberian husky and an Alaskan malamute. When researchers later measured how well a trained human annotator did on the same task, the answer was about 5 per cent, with the caveat that it was one person who had practised for weeks. The 2012 network was nowhere near that. By 2015 the residual networks of Kaiming He and colleagues at Microsoft Research had a top-5 error of 3.6 per cent on the same benchmark, and the challenge ended in 2017 because the task was considered solved.

The second thing the numbers meant was about the hand-designed features. For a decade computer vision had been the craft of designing descriptors (SIFT, HOG, and others) that captured the shape of an object in numbers, and then training a classifier on those numbers. AlexNet learned its own descriptors from pixels, and its first layer, printed in the paper as 96 small images, looked like the edge and colour detectors the craft had spent years approximating. Within two years almost every entry in the challenge was a convolutional network, and within five almost every paper in computer vision was.

Why 2012 and not 1990

It is worth asking what stopped this happening earlier, because the answer is the same thing that drove the rest of the decade. LeNet-5 had read digits in 1998 with the same kind of network. What it lacked was a million labelled photographs, a processor that could multiply matrices a hundred times faster, and the three training tricks. The ideas waited twenty years for the hardware and the data, which is a pattern the field has seen twice before, in the winters of the 1970s and the late 1980s, when the promises outran the computers.

It also explains what came next. If 2012 showed that a bigger network on more data with more compute beats a cleverer one, then the research programme writes itself: more data, more compute, bigger networks. The residual networks of 2015 were deeper; the transformers of 2017 were built to use the hardware better; the language models of 2020 were the same recipe applied to text at a scale that would have been absurd in 2012. Whether the recipe has a limit is one of the open questions of the field, and the people who disagree about it mostly agree about where it started.

The paper, "ImageNet Classification with Deep Convolutional Neural Networks", was published at the NeurIPS conference in December 2012 and is free to read. It is nine pages long and has been cited well over a hundred thousand times. Unit 8 of the AI Masterclass places the result in the history of the field's booms and winters, and Unit 9 explains the network layer by layer and what came after it.

Learn how machines learn, properly

The AI Masterclass: weekly units, quizzes, a final exam, and a certificate.
About the program