Does a language model know what it is talking about? Image: Paul_012 (CC BY-SA 3.0)

Does a language model know what it is talking about?

Here is a question that sounds philosophical and turns out to have an experiment. A language model is trained to predict the next token. Does it do that by storing statistics about which tokens follow which, a very large lookup table, or does it build inside itself a picture of the situation the text describes and predict from the picture? The two theories make the same predictions about the text. They make different predictions about what is inside the network, and in 2022 a group at Harvard and elsewhere found a case small enough to look.

The setup

Othello is played on an 8 by 8 board. Each move places a disc on a square and flips the opponent's discs that are sandwiched between it and another of the player's discs. A game is a list of about 60 squares. Kenneth Li, Martin Wattenberg, and colleagues trained a transformer of the kind that generates text, eight layers, on about 20 million synthetic games: sequences of legal moves, chosen at random, with no board, no rules, and no notion of a disc flipping. The model's only job was to predict the next move in the list. It learned to predict legal moves almost perfectly, with an error rate of a fraction of a per cent, on games it had not seen.

Predicting a legal move requires knowing which squares are empty and which discs have been flipped, which depends on the whole history. A lookup table of move sequences cannot do it; there are far too many sequences. So the model must be computing something about the board. The question was what, and whether it could be read.

The probe

A probe is a small classifier trained on the network's internal activations. Take the vector that the model holds at a middle layer after reading a partial game, and train a classifier to predict, from that vector alone, the state of one square: empty, black, or white. If the classifier succeeds, the information is in the vector in a form simple enough to extract. The authors trained a probe for each of the 64 squares. A linear probe, a single layer of weights, recovered the board poorly, with about 20 per cent of squares wrong. A probe with one hidden layer recovered it almost perfectly, with an error of under 2 per cent per square. The board was in there.

A probe on its own proves less than it seems to. The information might be present and unused, the way a photograph of a room contains the time on a clock that nobody looks at. The decisive experiment was the second one.

Editing the board

Using the probe, the authors changed the network's internal representation of the board. In the middle of a game, they took the activation vector and nudged it, by gradient steps against the probe, until the probe read one chosen square as flipped from black to white. Then they let the model continue and looked at the move it predicted. The predicted move changed to one that is legal on the edited board and not on the real one. The model was doing more than storing a board; its prediction depended on the stored board in the way that a player's would. Intervening on the inside changed the outside, which is the test that separates "contains" from "uses".

A year later Neel Nanda and others found that the board was in fact linearly readable after all, once the right coordinates were chosen: the model stores each square as mine or theirs, relative to the player about to move, rather than as black or white, and in that frame a single layer of weights reads it off. The finding says as much about the probe as about the model. A representation can look nonlinear because the question was asked in the wrong words.

What it shows and what it does not

The experiment shows that a model trained only on sequences, with next-token prediction as its only loss, can build an internal model of the process that generated the sequences, and can use it. That is a fact about the training procedure, and it transfers: there is nothing in it specific to Othello. It is evidence against the view that such models are only lookup tables. It settles the question for a board of 64 squares and 3 states.

It does not show that a model trained on the internet holds a model of the world. The world is not a board, its state is not discrete, and nobody has a probe for "what the text is about". What has been found in language models since are pieces: probes that read a place's longitude and latitude from the activations on its name, directions in the activation space for truth and falsehood, representations of the time of an event. Whether these pieces add up to understanding, and what understanding would mean of a system with no body and no stake in being right, is one of the disputed questions of the field. The positions are argued with evidence of this kind on both sides, and the masterclass presents them as disputed.

The method, probing and then intervening, is the method of interpretability research in general. It is how researchers found attention heads that copy repeated patterns, directions that make a model refuse, and features for individual concepts in the activations of models with billions of parameters, and it is how they test whether what they found is used. The tools of this work are now public; the Neuronpedia site lets anyone read the features of an open model and try the interventions.

The paper, "Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task", was presented at ICLR 2023 and is free on arXiv. Unit 16 of the AI Masterclass works a probe on a two-dimensional hidden vector by hand, explains superposition and sparse autoencoders, and goes on to the alignment questions that interpretability is meant to help answer.

Learn how machines learn, properly

The AI Masterclass: weekly units, quizzes, a final exam, and a certificate.
About the program