How machines learn
The ideas under every learning method: the loss, the gradient, one step downhill, and what goes wrong when the step is too large or the model too flexible.
Articles

Why bigger models kept getting better, and the rule for how big
In 2020 the loss of a language model was found to fall as a straight line on a log-log plot of compute, parameters, and data. In 2022 a 70 billion parameter model beat one four times its size by being trained on more text. The scaling laws, the Chinchilla rule, and what one training run costs in electricity.

Ten equations every AI student should know
The field runs on a small number of equations, and most are the same three ideas in different clothes: a dot product, a probability, a gradient. Ten of them, each in one line and one paragraph, from the first unit to the last.

Why a learning rate that is too large ruins everything
Four runs of gradient descent on the same bowl with learning rates of 0.05, 0.25, 0.75, and 1.1. One crawls, one converges, one oscillates, and one explodes, and the loss curve of each tells you which within ten steps.

What a gradient is and why every AI model follows one
Every model from a straight-line fit to a language model with hundreds of billions of parameters is trained by the same move: compute the slope of the loss and take a small step downhill. Three steps on a bowl, with every number, and the step that is too large.
In the masterclass
The AI Researcher's Toolkit
The vectors, probabilities, and derivatives that every later unit uses
Learning from Data
Linear regression, the loss function, gradient descent, and overfitting
Training Deep Networks
Stochastic gradient descent, learning rates, initialization, and the practice of training