Christopher D. Manning
Stanford University
- Research field
- Natural language processing, computational linguistics, and deep learning for language
Christopher Manning leads the Stanford natural language processing group and has worked for three decades on how machines can parse and understand human language; the dot-product attention score the transformer uses was proposed in a 2015 paper from his group. In a 2020 paper with Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy he asked whether a transformer trained only to fill in hidden words learns grammar on its own. By recording which word each of BERT's 144 attention heads looks at, and by fitting a linear map under which distances between word vectors match distances in a parse tree, the team showed that individual heads track relations such as a verb and its object, and that about 82 per cent of the edges of the syntax tree can be read from the vectors. The unit presents this work after the section on multi-head attention, as the answer to what the heads learn.
In the masterclass
Section 3, after multi-head attention, describes the attention heads of BERT that track grammatical relations and the structural probe that reads parse trees from its vectors.
The research is presented in Unit 11: Attention and Transformers, in a research spotlight box that explains the result at the level of the unit and links it to what you have just learned.