Language models and transformers
What a token is, what attention computes, how a text predictor becomes an assistant, and why bigger models kept getting better, with the rule for how big.
Articles

Does a language model know what it is talking about?
Train a small transformer on nothing but lists of moves from the board game Othello, with no picture of the board, and ask whether it holds the board in its head. A probe says yes; editing the board inside the network and watching the move change says it uses it.

How AlphaFold predicts the shape of a protein
A protein is a chain of amino acids that folds into a shape, and the shape is what it does. For fifty years predicting the shape from the chain was an open problem. In November 2020 a network did it to within the width of an atom, and in 2024 the Nobel Prize in Chemistry followed.

How a language model is turned into an assistant
A model that has read the internet continues text; it does not answer questions. Three stages turn it into an assistant: demonstrations, comparisons, and a reward with a leash. The InstructGPT numbers of 2022, and the Bradley-Terry probability worked for two answers.

Why bigger models kept getting better, and the rule for how big
In 2020 the loss of a language model was found to fall as a straight line on a log-log plot of compute, parameters, and data. In 2022 a 70 billion parameter model beat one four times its size by being trained on more text. The scaling laws, the Chinchilla rule, and what one training run costs in electricity.

What attention actually computes
Three tokens, four numbers each, three small matrices, and the formula that runs every transformer. Queries, keys, and values worked by hand to the output, and the reason the scores are divided by the square root of two.

What a token is and why your sentence costs more than mine
A language model does not read words. It reads tokens, pieces of text chosen by counting which pairs of characters occur together most often. Byte pair encoding on a corpus of five words, four merges, and why the same sentence costs more in Hindi than in English.
In the masterclass
Embeddings and Sequence Models
Words as vectors, recurrent networks, the LSTM, and language modelling
Attention and Transformers
Queries, keys, and values, self-attention, and the architecture of 2017
Pretraining and Scaling Laws
Next-token prediction, tokenization, compute, and the laws that relate loss to scale
Instruction Tuning and Learning from Human Feedback
From a text predictor to an assistant: demonstrations, preferences, and reward models
Reasoning, Tools, and Agents
Chains of thought, votes over samples, retrieved documents, tool calls, and models that act over many steps