← All research spotlights

Research spotlight · Unit 13

Sanmi Koyejo

Stanford University

Research field
Machine learning, trustworthy and reliable machine learning, the evaluation of large models

Sanmi Koyejo leads a group at Stanford University that studies how machine learning systems should be measured and when they can be trusted. With Rylan Schaeffer and Brando Miranda, also at Stanford, he asked whether the abilities that seem to appear suddenly in large language models are a property of the models or of the metrics used to score them. Their paper showed that a per-token improvement that is smooth and predictable turns into a sharp jump under any score that demands a completely right answer, that most reported emergent abilities were measured with two such scores, and that they soften or disappear under continuous ones. The unit presents the argument next to a drawing of its mathematical model and gives both readings of the evidence.

In the masterclass

Section 6, after the emergent abilities subsection, presents the paper's model of a smooth per-token improvement scored by a discontinuous metric and its three experiments.

The research is presented in Unit 13: Pretraining and Scaling Laws, in a research spotlight box that explains the result at the level of the unit and links it to what you have just learned.

The paper

Schaeffer, R., Miranda, B., and Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage?. Advances in Neural Information Processing Systems (NeurIPS 2023), 36. https://doi.org/10.48550/arXiv.2304.15004

The scientists named here are not contributors to the AI Masterclass, are not affiliated with it, and have not endorsed it. The program presents and explains their published research, with references to the original papers.