A data processing technician at a computer terminal in a Navy data centre in 1994, speaking on the telephone while typing, with dark equipment racks behind him
Unit 14 · Advanced + Large Language Models · 60 min

Instruction Tuning and Learning from Human Feedback

From a text predictor to an assistant: demonstrations, preferences, and reward models
Image: Department of Defense, American Forces Information Service, via Wikimedia Commons, public domain

A pretrained language model continues text; it does not answer. This unit follows the second phase of training that turns it into an assistant: supervised fine-tuning on written demonstrations, the Bradley-Terry model that turns pairwise comparisons into a reward model, reinforcement learning against that reward with a KL penalty as the leash, and direct preference optimization, which reaches the same goal without a reward model. It ends with what goes wrong (reward hacking, sycophancy, verbosity), how assistants are ranked by human votes, and whose preferences the whole procedure encodes.

Preview: the beginning of the unit

A pretrained language model is a very good predictor of text, and that is all it is. Given the line “How do I bake bread?”, the continuation it finds most likely, judging by the web pages it learned from, is another question such as “How long should the dough rise?”, because on the forums where such questions appear they come in lists. Between that predictor and an assistant that answers the question lies a second phase of training. It uses a small fraction of one per cent of the data of pretraining, and it decides almost everything the reader notices when talking to the model. This unit follows that phase in the order in which the field built it. Demonstrations come first, comparisons replace them, a reward model and reinforcement learning turn the comparisons into training, a shortcut removes the reward model again, and the unit closes with what the procedure optimizes and who decides that.

1 Why a predictor is not an assistant

Unit 13 described pretraining: the model reads trillions of tokens of text and is trained, with the cross-entropy of Unit 5, to put a high probability on the token that came next. The loss is minimized by a model that imitates the whole distribution of its training text. That text contains answers, but it also contains questions without answers, advertisements, arguments, fiction, and confident statements that are false, and the model has learned to continue each of them in the same style. Three consequences follow for anyone who types a request into such a model.

  • It continues; it does not answer. A question is followed by whatever usually follows a question on the web, which is often another question or a line of a dialogue, as the example of the opening paragraph shows.
  • It imitates average text. An answer written by the model is as reliable as the typical page, and the typical page is not written by an expert.
  • It has no notion of who is asking or what the person wants. A pretrained model will complete a request for medical advice with a forum thread and a request for a poem with a comment on the poem.

The few-shot prompting of Unit 13 helps with the first point, because a prompt that shows two answered questions makes a third answer the most likely continuation. It does not help much with the other two, and it costs part of the context on every request. Table 1 shows the difference between a pretrained model and the same model after the training of this unit, on a prompt from the paper that introduced the method for assistants.

SourceText
PromptSerendipity means the occurrence and development of events by chance in a happy or beneficial way. Use the word in a sentence.
GPT-3Serendipity is the ability to see something good in something bad. Use the word in a sentence. [The model continues with two more such lines.]
InstructGPTSerendipity can be defined as the happy chance occurrence of events leading to a beneficial outcome. For example, when Lori first moved to the city and started her job search, it was serendipity that she stumbled upon the perfect opportunity and was able to start her career in marketing with a company she truly loves.
Table 1. A prompt written by a labeller and the completions of the 175 billion parameter GPT-3 and of the InstructGPT model tuned from it, from Figure 47 of the 2022 paper by Ouyang and colleagues at OpenAI (arXiv:2203.02155), shortened where marked. The pretrained model imitates a worksheet with several such exercises; the tuned model does the exercise.

1.1 Supervised fine-tuning on demonstrations

The direct remedy is to show the model what an answer looks like. People write a set of prompts and, for each prompt, the answer they would want, and the model is trained on these pairs. This is supervised fine-tuning, abbreviated SFT, and the pairs are called demonstrations. The training is ordinary next-token prediction, with one change: the prompt is given to the model as context, but the loss counts only the tokens of the answer. A model scored on the prompt tokens as well would learn to write questions in the style of the users, which is what the pretrained model already does.

The InstructGPT project at OpenAI, published in 2022 by Long Ouyang and 19 colleagues, collected about 13,000 such prompts with written demonstrations and trained GPT-3 on them for 16 passes. A 2023 paper from Meta, called LIMA, went further in the same direction and tuned a 65 billion parameter model on only 1,000 carefully written demonstrations, with no further training, and found that people rated its answers as well as or better than those of several assistants trained with far more data. Only GPT-4 stayed clearly ahead of it. The authors drew the conclusion that almost all of the knowledge and ability of an assistant comes from pretraining, and that fine-tuning mainly teaches the model which part of what it knows to show and in what form.

The full unit is part of the program

Want to see a complete unit first? Unit 3, Reasoning under Uncertainty, is free to read.