← All research spotlights

Research spotlight · Unit 14

Chelsea Finn

Stanford University

Research field
Machine learning for robots and language models, learning from preferences

Chelsea Finn is a computer scientist at Stanford University whose group works on how robots and language models learn from experience and from human feedback. In a 2023 paper with Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, and Christopher Manning, the group showed that the reward model and the reinforcement learning loop of RLHF can be removed: the optimal policy of the KL-regularized objective has a closed form, so the reward can be written through the policy and the comparisons can train the language model directly with one classification loss, a method they named direct preference optimization. On summarization and dialogue it matched or exceeded PPO-based RLHF at a fraction of the engineering cost. The unit derives the loss from the reward model loss and presents the paper's experiments after the section on RLHF.

In the masterclass

Section 4, after the derivation of the DPO loss, presents the paper's three experiments and what the method changed.

The research is presented in Unit 14: Instruction Tuning and Learning from Human Feedback, in a research spotlight box that explains the result at the level of the unit and links it to what you have just learned.

The paper

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems (NeurIPS 2023), 36, arXiv:2305.18290. https://doi.org/10.48550/arXiv.2305.18290

The scientists named here are not contributors to the AI Masterclass, are not affiliated with it, and have not endorsed it. The program presents and explains their published research, with references to the original papers.