Interpretability and Alignment
A trained network is a list of numbers, and this closing unit asks what they mean and whether the system they define does what its users intend. It works through linear probes and the Othello experiment, in which a model trained only on move sequences turned out to hold the board in its activations and to play from it, then through superposition, sparse autoencoders, activation patching, induction heads, and how far circuits have been mapped. The second half defines alignment, separates outer from inner alignment with published cases of specification gaming and hidden behaviour, explains why jailbreaks work and why scalable oversight is open, and gives the policy record of 2023 to 2026 as fact, ending with the open problems of the field and how to read a research paper.
A trained network is a list of numbers. Unit 6 showed how the numbers are found and Unit 13 how many of them a large model has, but neither unit could say what any one of them does, and a model that answers correctly for reasons nobody can state is a model nobody can fully trust. This last unit takes up the two questions that follow from that. The first is interpretability, the attempt to read the computation out of the weights and activations of a trained network. The second is alignment, the attempt to make a system do what its users and operators intend rather than what a loophole in its training rewards. Both are young fields with more open problems than settled results, and the unit tries to say at each point which is which.
1 Three questions about a trained network
Unit 5 drew a line between a model a person can read in full, such as a decision tree of depth three, and a black box, a model whose prediction is right or wrong without anyone being able to say why. A network with billions of weights is the second kind, and interpretability research asks whether it has to stay that way. The questions it puts to a trained network come in three sorts.
- What does a part of the network represent? Does one direction through the components of a hidden vector stand for something a person can name, such as “this square of the board is empty” or “the text is in Arabic”?
- Which parts cause which behaviour? If a model gives the right answer, which neurons, heads, and layers did the work, and would the answer change if they were switched off?
- Does the model hold a model of the world? A system trained only to predict the next token could memorize surface statistics of its training text, or it could build an internal picture of whatever produced the text and compute from that. The unit’s first experiment tells the two apart.
The branch of the field that tries to answer such questions by reverse-engineering the computation, component by component, calls itself mechanistic interpretability. Its methods divide into two kinds by the evidence they give. A method that reads activations and finds a pattern shows that information is present, which is a correlation; a method that changes activations and watches the output shows that the information is used, which is a causal claim. The sections that follow present the methods in that order.
One piece of vocabulary is needed throughout the unit. Unit 11 built the transformer from blocks that each add their output to the vector they received, through the residual connection of Unit 7. The vector that passes through all the blocks at one token position is written \(\mathbf{h}^{(l)}\) in this unit when it is read after block \(l\), and the field calls the sequence of these vectors the residual stream, because every block reads from it and writes into it. “The activations at layer \(l\)” means \(\mathbf{h}^{(l)}\) at some or all positions.