← All research spotlights

Research spotlight · Unit 16

Martin Wattenberg

Harvard University

Research field
Interpretability of neural networks, visualization of machine learning

Martin Wattenberg works on ways of seeing what a neural network has learned, from visualizations of embeddings to probes and interventions on a model's internal activations. With Kenneth Li, Aspen Hopkins, David Bau, Fernanda Viégas, and Hanspeter Pfister he trained a small transformer on nothing but sequences of Othello moves and showed that it builds a picture of the board inside its activations: nonlinear probes read the board with an error of 1.7 per cent, and editing the activations until the probe reports one square flipped changes the model's legal-move predictions to those of the edited board. The unit presents this experiment as its worked case of probing and intervention, together with the follow-up paper in which a linear probe succeeds once the board is encoded relative to the player about to move.

In the masterclass

Section 2, after the probing subsection, presents the Othello model, the probe results, and the intervention test.

The research is presented in Unit 16: Interpretability and Alignment, in a research spotlight box that explains the result at the level of the unit and links it to what you have just learned.

The paper

Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., and Wattenberg, M. (2023). Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. International Conference on Learning Representations (ICLR 2023). https://doi.org/10.48550/arXiv.2210.13382

The scientists named here are not contributors to the AI Masterclass, are not affiliated with it, and have not endorsed it. The program presents and explains their published research, with references to the original papers.