How a language model is turned into an assistant Image: Unknown authorUnknown author or not provided, public domain

How a language model is turned into an assistant

Give a model that has only been trained to predict the next token the prompt "How do I bake bread?" and the most likely continuation, by the statistics of the web, is more questions: "How long does it take? What flour should I use?" That is what a pretrained language model is, a predictor of text, and a very good one. Turning it into something that answers took three further stages, described in a paper from OpenAI in March 2022 on a system called InstructGPT, and the numbers in that paper are the best public account of what it costs.

Stage one: show it what an answer looks like

The first stage is supervised fine-tuning. About 40 contractors wrote demonstrations: for a prompt, the answer a helpful assistant should give. The InstructGPT team collected about 13,000 such prompts with answers and continued training the pretrained model on them with the same next-token loss it had always used, except that now the text it was predicting was the answer and the prompt was given. After this the model answers questions. It also inherits every habit of the demonstrators, writes at the length they wrote, and cannot do better than them, because it was trained to imitate.

Stage two: comparisons, not scores

Writing the best possible answer is slow and skilled. Saying which of two answers is better is fast and most people agree about it most of the time. So the second stage collected comparisons: for a prompt, several answers from the fine-tuned model, and a labeller's ranking of them. The team gathered about 33,000 prompts' worth of rankings. These train a reward model, a copy of the language model with its output replaced by a single number \(r(x, y)\), the score of answer \(y\) to prompt \(x\).

The link between a score and a comparison is the Bradley-Terry model, which was proposed in 1952 for ranking chess players and sports teams from match results. It says that the probability that answer 1 beats answer 2 is the sigmoid of the difference of their scores:

\[ P(y_1 \succ y_2) = \sigma\big(r(y_1) - r(y_2)\big) = \frac{1}{1 + e^{-(r(y_1) - r(y_2))}}. \]

If the reward model gives the two answers scores of 2.0 and 1.0, it predicts that the first is preferred with probability \(\sigma(1) = 0.73\). If it gives them 2.0 and 1.9, the probability is \(\sigma(0.1) = 0.52\), a coin toss. The reward model is trained by gradient descent to make the labellers' actual choices probable: for each pair where the labeller preferred \(y_w\) to \(y_l\), minimize \(-\log \sigma(r(y_w) - r(y_l))\), which is the cross-entropy loss of Unit 5 in another costume. The scores it learns have no units and no zero; only differences matter, exactly as in chess ratings.

Stage three: a reward with a leash

Now the language model is trained to produce answers that the reward model scores highly. This is reinforcement learning: the model's answer is an action, the reward model's score is the reward, and the policy gradient methods of Unit 12 (the paper used one called PPO) raise the probability of answers that scored well. About 31,000 prompts were used for this stage. The objective has a second term:

\[ \mathbb{E}\big[r(x, y)\big] - \beta\, \mathrm{KL}\big(\pi_{\boldsymbol{\theta}} \,\|\, \pi_{\mathrm{ref}}\big). \]

The Kullback-Leibler divergence measures how far the tuned model's distribution over answers has drifted from the fine-tuned model it started from, and \(\beta\) sets the strength of the penalty. Without it the model finds answers that the reward model loves and no human would, strings of flattering phrases, lists of hedges, text that is not quite language. The reward model is a learned approximation of human judgement, and like any learned model it is wrong off its training distribution; the leash keeps the policy on the distribution where the reward is trustworthy. Reward hacking, when the leash is too loose, is the field's name for the failure.

What the numbers said

The headline result was that labellers preferred the answers of the 1.3 billion parameter InstructGPT to those of the 175 billion parameter GPT-3 it was built from, a model more than a hundred times larger. Three stages on tens of thousands of examples had done what scale alone had not, which is to point the model's abilities at the task. The tuned models were also truthful somewhat more often and produced toxic text somewhat less often on the benchmarks the paper reports, with small losses on some academic benchmarks that the team called an alignment tax and partly recovered by mixing pretraining data back into the last stage.

The method's origin is a 2017 paper by Paul Christiano and colleagues, in which a simulated figure learned to do a backflip from about 900 comparisons made by a human watching pairs of short clips, in under an hour of the human's time. Writing a reward function for a backflip is nearly impossible; saying which of two attempts looks more like one is easy. The same asymmetry is what makes comparisons the right data for "helpful, honest, and harmless", which nobody can write down as a formula either.

Since then

In 2023 Rafael Rafailov, Chelsea Finn, and colleagues at Stanford showed that the reward model and the reinforcement learning loop could be removed entirely. Their direct preference optimization rewrites the objective above so that the language model itself plays the role of the reward model, and the comparisons train it directly with a loss that looks like the Bradley-Terry one. It reaches similar results with far less machinery and is now the common choice for open-weight models. Other laboratories replaced some human comparisons with comparisons made by a model following a written set of principles, and the assistants of 2024 and 2025 added a further stage in which the model is trained by reinforcement on problems with checkable answers, such as mathematics and code.

How assistants are judged against each other follows the same logic as how they are trained. Public arenas show two anonymous answers to the same prompt and ask the user which is better, and the votes are turned into ratings by the Bradley-Terry model again, on the Elo scale from chess. A rating gap of 100 points means the higher-rated model wins about 64 per cent of comparisons; a gap of 400 points, about 91 per cent. A few hundred votes give a rating with an uncertainty of tens of points, so the small gaps at the top of a leaderboard are often within the noise.

The papers are Ouyang et al. (2022) on InstructGPT, Christiano et al. (2017) on learning from preferences, and Rafailov et al. (2023) on DPO, all free on arXiv. Unit 14 of the AI Masterclass computes the KL divergence between two distributions by hand, derives the DPO loss from the reward model's, and gives the InstructGPT numbers with their sources; Unit 16 takes up what "aligned" should mean.

Learn how machines learn, properly

The AI Masterclass: weekly units, quizzes, a final exam, and a certificate.
About the program