Rows of drums on a rebuilt Turing Bombe at Bletchley Park, seen from above, each drum carrying rings of contacts
Unit 15 · Advanced + Large Language Models · 60 min

Reasoning, Tools, and Agents

Chains of thought, votes over samples, retrieved documents, tool calls, and models that act over many steps
Image: Mike Peel (www.mikepeel.net), via Wikimedia Commons, CC BY-SA 4.0

An assistant answers in one pass, with a fixed amount of computation per token and no way to look anything up or act. This unit loosens those constraints one at a time: writing the steps before the answer, sampling several chains and letting them vote, training on problems with checkable answers, retrieving documents into the prompt, calling tools, and running the model in a loop of thought, action, and observation, which is what the field now calls an agent. It closes with what agent benchmark scores mean, why several language models are not yet a multi-agent system, and the failure modes a reader who relies on these systems must know, prompt injection first among them.

Preview: the beginning of the unit

The assistant of Unit 14 answers in one pass. It reads the prompt, and every token it writes is one forward pass through the same stack of layers, with no pause to think, no scratch paper, no way to look anything up, and no way to act. Everything in this unit loosens one of those constraints. A model that writes its steps before its answer gets more computation and a memory; a model that is sampled several times and made to vote gets a second opinion; a model trained on problems with checkable answers learns which steps lead somewhere; a model that is handed retrieved documents and tools gets facts it never saw and arithmetic it cannot do; and a model run in a loop of thought, action, and observation becomes what the field now calls an agent. The unit closes with the ways in which such systems fail, because a reader who will rely on them needs the failure modes more than the demonstrations.

1 Reasoning one token at a time

The transformer of Unit 11 computes every output token with the same, fixed amount of work. A model with 96 layers spends 96 layers on the token after “The capital of France is” and 96 layers on the first digit of the answer to a three-digit multiplication, although the first task needs a lookup and the second needs a dozen dependent steps of arithmetic. Within one token the model cannot iterate, that is, it cannot repeat a step as many times as the problem needs. What it can do is write, and every token it writes becomes part of the context that the next forward pass reads. A written intermediate result therefore serves as extra computation, because producing it took a whole forward pass, and it also serves as memory, because all later tokens can attend to it through the attention of Unit 11. Writing the steps turns a single pass into a sequence of passes that build on each other.

This is the mechanism behind chain-of-thought prompting, published in January 2022 by Jason Wei and colleagues at Google. In a few-shot prompt of the kind Unit 13 described, each of the example questions was followed by a few sentences of worked reasoning before its answer, and the model imitated the form, writing its own reasoning before its own answer. On GSM8K, a set of 8,500 school mathematics word problems, Google’s PaLM model with 540 billion parameters solved 17.9 per cent of the test problems when prompted with questions and bare answers and 56.9 per cent when prompted with eight questions and their worked reasoning. The gain appeared only in large models; below about 100 billion parameters the written chains were fluent and wrong, and accuracy fell. A paper from the University of Tokyo and Google four months later, by Takeshi Kojima and colleagues, found that no examples were needed at all. Appending the words “Let’s think step by step” after the question raised the accuracy of the InstructGPT model of Unit 14 on another arithmetic set, MultiArith, from 17.7 to 78.7 per cent, and on GSM8K from 10.4 to 40.7 per cent.

A direct answer and a chain of thought for the same question. In the direct answer the digits are produced one after another with nothing to build on…
Figure 1. A direct answer and a chain of thought for the same question. In the direct answer the digits are produced one after another with nothing to build on. In the chain, each written step is read by the forward passes that produce the following steps, so the model computes in several passes instead of one.

Two cautions belong next to this result. A written chain that reaches a wrong answer looks exactly like one that reaches a right answer; the model imitates the form of reasoning, and nothing forces the content to be sound. And the chain is not always the reason for the answer. A 2023 study by Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman at New York University found that when the examples in a few-shot prompt were arranged so that the correct option was always (A), the models drifted toward answering (A) on new questions and wrote plausible chains that justified the choice without mentioning the pattern that had caused it. The chain improves the answer, and it is a record that may or may not be faithful, that is, a true account of what produced the answer; Unit 16 returns to what a model’s account of itself is worth. The gains are also sensitive to wording and to the order of the examples, which is why the next section replaces one chain by many.