← All research spotlights

Research spotlight · Unit 11

Christopher D. Manning

Stanford University

Research field
Natural language processing, computational linguistics, and deep learning for language

Christopher Manning leads the Stanford natural language processing group and has worked for three decades on how machines can parse and understand human language; the dot-product attention score the transformer uses was proposed in a 2015 paper from his group. In a 2020 paper with Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy he asked whether a transformer trained only to fill in hidden words learns grammar on its own. By recording which word each of BERT's 144 attention heads looks at, and by fitting a linear map under which distances between word vectors match distances in a parse tree, the team showed that individual heads track relations such as a verb and its object, and that about 82 per cent of the edges of the syntax tree can be read from the vectors. The unit presents this work after the section on multi-head attention, as the answer to what the heads learn.

In the masterclass

Section 3, after multi-head attention, describes the attention heads of BERT that track grammatical relations and the structural probe that reads parse trees from its vectors.

The research is presented in Unit 11: Attention and Transformers, in a research spotlight box that explains the result at the level of the unit and links it to what you have just learned.

The paper

Manning, C. D., Clark, K., Hewitt, J., Khandelwal, U., Levy, O. (2020). Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48), 30046‑30054. https://doi.org/10.1073/pnas.1907367117

The scientists named here are not contributors to the AI Masterclass, are not affiliated with it, and have not endorsed it. The program presents and explains their published research, with references to the original papers.