How to read an AI paper as a beginner Image: Jérémy Barande (CC BY-SA 3.0)

How to read an AI paper as a beginner

Most papers in this field are posted on arXiv, a free preprint server, before or instead of being published in a journal, and anyone can read them. Few beginners do, because a paper is written for the people who already know the field, and it is not printed in the order in which it should be read. Here is the order, with one paper as the example: "Deep Residual Learning for Image Recognition" by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, posted in December 2015, which introduced the residual networks that most networks since are built on.

First the abstract, and one question

Read the abstract and write down, in a sentence, what the paper claims. For the residual networks paper: deeper networks had been getting harder to train, and a "shortcut" that adds a layer's input to its output lets networks of 152 layers train and win the 2015 ImageNet competition. The one question to hold for the rest of the reading is: what would convince me that this claim is true? For a claim about training, the answer is a comparison on the same data between the method and a baseline that differs only in the thing claimed. Keep that in mind; the paper may or may not provide it.

Then the figures

Go through every figure before reading any method. Figures are where the authors put the evidence they are proudest of, and a figure read carefully tells you most of what the paper found. Figure 1 of the residual networks paper shows training error and test error against the number of training iterations for a 20-layer and a 56-layer network of the plain kind, on a dataset of small images. The 56-layer network has higher error on both. Read the axes: higher training error as well as higher test error. That rules out overfitting as the explanation, because an overfitting network would have lower training error. Something is stopping the deeper network from fitting even the training data, which the authors call degradation, and the paper is about fixing it. That one figure is the problem statement.

Look at what is on the axes, whether the axes are logarithmic, whether the error bars exist, and how many runs the curves average. Many figures in this field have no error bars, which means the result might not survive a change of random seed; it is a thing to notice, not a reason to stop reading.

Then the main table

Find the table with the headline numbers. In this paper it is the table of ImageNet results, with the top-1 and top-5 error of each architecture on the validation set. Read down it: the plain 18-layer and 34-layer networks, where the deeper one is worse; the residual 18-layer and 34-layer, where the deeper one is better; then 50, 101, and 152 layers, each better again. At the end, an ensemble of residual networks with a top-5 error of 3.57 per cent on the test set, which won the competition. That table is the comparison you wrote down as the one that would convince you: the same data, plain against residual at the same depth, and the difference attributable to the shortcut.

Check what the baseline is and whether it is a fair one. Here the plain network at the same depth is the right baseline, and the authors say it has the same number of parameters. A paper that compares its method against a baseline from an older paper, trained by other people on other hardware with other settings, is making a weaker claim than it looks.

Then the ablations

An ablation removes one piece of the method at a time to see what each piece contributes. It is where a paper is most honest, and where a beginner learns most, because it says which of the ideas mattered. The residual networks paper compares three ways of making the shortcut (identity, identity with zero padding, and a learned projection) and finds they perform similarly, which tells you that the shortcut matters and its exact form does not. A paper without ablations has told you that a bundle of ideas works and has not told you which ones.

Where the caveats hide

They hide in four places. In the training details, usually an appendix: the learning rate schedule, the data augmentation, the number of GPUs, and the hours, which say what it costs to reproduce the result and whether the baseline got the same care. In the choice of benchmark: a method that wins on the authors' benchmark and is not tested elsewhere may be tuned to it. In the sentences with "we note that" and "we leave to future work", which mark the things the authors know are missing. And in what is not compared: if a natural baseline is absent from the table, ask why. None of these is dishonesty; the authors have a page limit and a deadline. Reading them is the difference between knowing what a paper found and knowing what it showed.

Only then the method

With the claim, the evidence, and the caveats in hand, read the method section, and read it with a pencil. For the residual networks paper, the method is one equation: a block computes \(\mathbf{h}^{(l+1)} = \mathbf{h}^{(l)} + F(\mathbf{h}^{(l)})\), the input plus a learned function of the input, instead of \(F(\mathbf{h}^{(l)})\) alone. The authors' argument for it fits in a paragraph: if the best thing a layer could do were nothing, a plain layer would have to learn the identity map, which is hard, while a residual layer learns to output zero, which is easy. Whether that argument is the real reason is debated to this day; the result is not.

Practical habits

Read the paper twice, once fast for the shape and once slow for the method. Look up the two or three cited papers that the method depends on and read their abstracts. Keep a file of one-paragraph summaries, in your own words, with the claim, the evidence, and the caveat for each paper. Use Semantic Scholar or Google Scholar to see who has cited a paper since, because the follow-up papers often contain the replication or the correction. And when a paper's main result is one you can reproduce on a laptop in an afternoon, do it; nothing teaches a method like watching it fail on your first attempt.

The paper is free on arXiv. Unit 7 of the AI Masterclass presents the residual connection as the device that made deep networks trainable, with Kaiming He's own account in its research spotlight, and every unit's literature list names the free copy of each paper it cites so that this habit can start in week one.

Learn how machines learn, properly

The AI Masterclass: weekly units, quizzes, a final exam, and a certificate.
About the program