Sanmi Koyejo
Stanford University
- Research field
- Machine learning, trustworthy and reliable machine learning, the evaluation of large models
Sanmi Koyejo leads a group at Stanford University that studies how machine learning systems should be measured and when they can be trusted. With Rylan Schaeffer and Brando Miranda, also at Stanford, he asked whether the abilities that seem to appear suddenly in large language models are a property of the models or of the metrics used to score them. Their paper showed that a per-token improvement that is smooth and predictable turns into a sharp jump under any score that demands a completely right answer, that most reported emergent abilities were measured with two such scores, and that they soften or disappear under continuous ones. The unit presents the argument next to a drawing of its mathematical model and gives both readings of the evidence.
In the masterclass
Section 6, after the emergent abilities subsection, presents the paper's model of a smooth per-token improvement scored by a discontinuous metric and its three experiments.
The research is presented in Unit 13: Pretraining and Scaling Laws, in a research spotlight box that explains the result at the level of the unit and links it to what you have just learned.