Blog topic

Language models and transformers

What a token is, what attention computes, how a text predictor becomes an assistant, and why bigger models kept getting better, with the rule for how big.

Articles

Does a language model know what it is talking about?
3 Nov 2026

Does a language model know what it is talking about?

Train a small transformer on nothing but lists of moves from the board game Othello, with no picture of the board, and ask whether it holds the board in its head. A probe says yes; editing the board inside the network and watching the move change says it uses it.

How AlphaFold predicts the shape of a protein
28 Oct 2026

How AlphaFold predicts the shape of a protein

A protein is a chain of amino acids that folds into a shape, and the shape is what it does. For fifty years predicting the shape from the chain was an open problem. In November 2020 a network did it to within the width of an atom, and in 2024 the Nobel Prize in Chemistry followed.

How a language model is turned into an assistant
22 Oct 2026

How a language model is turned into an assistant

A model that has read the internet continues text; it does not answer questions. Three stages turn it into an assistant: demonstrations, comparisons, and a reward with a leash. The InstructGPT numbers of 2022, and the Bradley-Terry probability worked for two answers.

Why bigger models kept getting better, and the rule for how big
16 Oct 2026

Why bigger models kept getting better, and the rule for how big

In 2020 the loss of a language model was found to fall as a straight line on a log-log plot of compute, parameters, and data. In 2022 a 70 billion parameter model beat one four times its size by being trained on more text. The scaling laws, the Chinchilla rule, and what one training run costs in electricity.

What attention actually computes
28 Sep 2026

What attention actually computes

Three tokens, four numbers each, three small matrices, and the formula that runs every transformer. Queries, keys, and values worked by hand to the output, and the reason the scores are divided by the square root of two.

What a token is and why your sentence costs more than mine
22 Sep 2026

What a token is and why your sentence costs more than mine

A language model does not read words. It reads tokens, pieces of text chosen by counting which pairs of characters occur together most often. Byte pair encoding on a corpus of five words, four merges, and why the same sentence costs more in Hindi than in English.