Image: Derrick Coetzee from Berkeley, CA, USA (CC0)Why bigger models kept getting better, and the rule for how big
Between 2018 and 2020 the largest language models grew from about 100 million parameters to 175 billion, a factor of more than a thousand, and at every step the bigger model was better. Nobody had a theory that said this should continue. What the field had instead, from 2020, was a measurement: plot the loss against the compute, the parameters, or the data on logarithmic axes and the points fall on a straight line over many orders of magnitude. A straight line on a log-log plot is a power law, and a power law can be extrapolated. That measurement is the reason the models kept growing.
The power laws of 2020
Jared Kaplan, Sam McCandlish, and colleagues at OpenAI trained transformers of many sizes on text and fitted the test loss (the cross-entropy per token, in nats) as a function of three things, varying one while the others were ample. Against parameters \(N\), the loss fell as \(N^{-0.076}\). Against dataset size \(D\), as \(D^{-0.095}\). Against compute \(C\), as about \(C^{-0.05}\). The exponents are small: multiplying compute by ten lowers the loss by about 11 per cent of its value. But the lines were straight across seven orders of magnitude of compute, and that regularity was the surprise. The architecture mattered little within the transformer family; the width and depth could be traded over a wide range; and the loss depended on the totals.
The 2020 paper also drew a conclusion about how to spend a budget: the loss fell faster with parameters than with data, so for a fixed amount of compute, the best plan was a very large model trained on relatively little text, stopping well before convergence. GPT-3, trained the same year with 175 billion parameters on about 300 billion tokens, followed that recipe.
The Chinchilla correction
In 2022 Jordan Hoffmann and colleagues at DeepMind repeated the experiment with the learning rate schedule set properly for each training length (the 2020 runs had used one schedule for all, which penalized the shorter ones) and found a different optimum. They trained more than 400 models, fitted the loss as
\[ L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} \]with \(E = 1.69\), \(A = 406.4\), \(B = 410.7\), \(\alpha = 0.34\), and \(\beta = 0.28\), and minimized it under the constraint \(C = 6ND\). The answer: parameters and tokens should grow at the same rate, and the optimal number of tokens is about twenty times the number of parameters. By that rule GPT-3 should have been trained on about 3.5 trillion tokens, not 300 billion. The team then trained Chinchilla, a 70 billion parameter model, on 1.4 trillion tokens, the same compute as their earlier 280 billion parameter Gopher trained on 300 billion, and Chinchilla was better on almost every benchmark. Smaller model, four times the data, same cost.
The fit also gives numbers. Put GPT-3's \(N\) and \(D\) into the formula and the predicted loss is 2.00 nats per token; put Chinchilla's in and it is 1.94. For a budget of \(10^{23}\) FLOP, the optimum under the 20-to-1 rule is about 29 billion parameters on about 580 billion tokens, with a predicted loss of 2.01. The constant \(E = 1.69\) is the loss the formula says no model could beat, the irreducible entropy of the text under this family of models; whether that floor is real is an open question.
What a run costs
The constraint \(C = 6ND\) is where the budgets come from. A forward pass costs about 2 floating point operations per parameter per token (a multiply and an add for each weight) and the backward pass about twice that, so 6 in all. For GPT-3, \(6 \times 175 \times 10^9 \times 300 \times 10^9 = 3.15 \times 10^{23}\) FLOP, which matches the \(3.14 \times 10^{23}\) in the paper's own table. An NVIDIA A100 GPU of 2020 sustained perhaps \(10^{14}\) FLOP per second on this kind of work, so the run was about \(3 \times 10^9\) GPU-seconds: 1,000 GPUs for about 36 days. The actual training used a cluster of V100 GPUs over a longer time, but the arithmetic is the right size.
Energy follows from the hours. A 2021 study by David Patterson and colleagues at Google and Berkeley estimated GPT-3's training at about 1,287 megawatt-hours. A European household uses about 3.5 megawatt-hours a year, so one training run was about 370 household-years of electricity, or the annual consumption of a village. Runs since then are reported, where they are reported at all, at ten to a hundred times that compute, and the energy and the carbon of training have become a research subject of their own. The energy of answering queries once the model is deployed, multiplied by billions of queries, is larger still.
What the laws do not say
The scaling laws are empirical. They describe how the loss of one family of models trained one way on one kind of data fell as the inputs grew, across the range that was measured, and they were revised once already when a schedule was fixed. They predict the loss, a number of nats per token, and not what the model can do with it; the relation between the loss and the ability to answer a question or write a program is the subject of a different argument, about whether abilities appear suddenly at scale or only look as if they do because of how they are scored. And they say nothing about where the data will come from when the 20-to-1 rule asks for more text than has been written, which by some estimates was reached around 2024 for public text of good quality, and is why so much current work is about data quality, synthetic data, and training for longer on the same text.
The two papers are Kaplan et al. (2020) and Hoffmann et al. (2022), both free on arXiv, and the energy estimate is in Patterson et al. (2021). Unit 13 of the AI Masterclass derives the 6ND rule from Unit 7's cost of a training step, plots the Chinchilla fit, works the optimum for a budget by hand, and presents both sides of the argument about emergent abilities with the paper that started it.


