Chelsea Finn
Stanford University
- Research field
- Machine learning for robots and language models, learning from preferences
Chelsea Finn is a computer scientist at Stanford University whose group works on how robots and language models learn from experience and from human feedback. In a 2023 paper with Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, and Christopher Manning, the group showed that the reward model and the reinforcement learning loop of RLHF can be removed: the optimal policy of the KL-regularized objective has a closed form, so the reward can be written through the policy and the comparisons can train the language model directly with one classification loss, a method they named direct preference optimization. On summarization and dialogue it matched or exceeded PPO-based RLHF at a fraction of the engineering cost. The unit derives the loss from the reward model loss and presents the paper's experiments after the section on RLHF.
In the masterclass
Section 4, after the derivation of the DPO loss, presents the paper's three experiments and what the method changed.
The research is presented in Unit 14: Instruction Tuning and Learning from Human Feedback, in a research spotlight box that explains the result at the level of the unit and links it to what you have just learned.