What attention actually computes Image: Jkarjalainen (CC BY-SA 3.0)

What attention actually computes

Attention is a soft dictionary lookup. A dictionary maps a key to a value: you bring a query, find the key that matches it, and take the value stored there. Attention does the same with vectors, with one change: instead of taking the value of the single matching key, it takes a weighted average of all the values, weighted by how well each key matches the query. That is the whole idea. The formula, the heads, and the transformer are what you get when you write it down and repeat it.

Three tokens

Suppose a sequence of three tokens, each represented by a four-component vector (a real model uses hundreds or thousands of components). Stack them as the rows of a 3 by 4 matrix \(\mathbf{X}\):

\[ \mathbf{X} = \begin{pmatrix} 1 & 0 & 1 & 0 \\ 0 & 1 & 0 & 1 \\ 1 & 1 & 0 & 0 \end{pmatrix}. \]

Attention needs three views of each token: a query (what am I looking for?), a key (what do I offer?), and a value (what do I pass on if chosen?). Each is produced by multiplying \(\mathbf{X}\) by a learned matrix. Take 4 by 2 matrices, so that the queries, keys, and values have two components each (the key dimension \(d_k\) is 2):

\[ \mathbf{W}_Q = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 0 \\ 0 & 1 \end{pmatrix}, \quad \mathbf{W}_K = \begin{pmatrix} 0 & 1 \\ 1 & 0 \\ 1 & 0 \\ 0 & 1 \end{pmatrix}, \quad \mathbf{W}_V = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \\ 0 & 0 \end{pmatrix}. \]

These numbers are set by hand for the example; in a trained model they are learned by gradient descent like every other weight. Multiplying gives the queries \(\mathbf{Q} = \mathbf{X}\mathbf{W}_Q\), the keys \(\mathbf{K} = \mathbf{X}\mathbf{W}_K\), and the values \(\mathbf{V} = \mathbf{X}\mathbf{W}_V\), each a 3 by 2 matrix with one row per token:

\[ \mathbf{Q} = \begin{pmatrix} 2 & 0 \\ 0 & 2 \\ 1 & 1 \end{pmatrix}, \quad \mathbf{K} = \begin{pmatrix} 1 & 1 \\ 1 & 1 \\ 1 & 1 \end{pmatrix}, \quad \mathbf{V} = \begin{pmatrix} 2 & 1 \\ 0 & 1 \\ 1 & 1 \end{pmatrix}. \]

Check one entry: the first row of \(\mathbf{X}\) is (1, 0, 1, 0), and its dot product with the first column of \(\mathbf{W}_Q\), which is (1, 0, 1, 0), is 2. That is the top left entry of \(\mathbf{Q}\).

Scores, weights, output

The score of token \(i\) looking at token \(j\) is the dot product of query \(i\) with key \(j\). All nine scores at once are \(\mathbf{Q}\mathbf{K}^{\top}\), a 3 by 3 matrix. Here every key is (1, 1), so the score of query (2, 0) with every key is 2, of query (0, 2) is 2, and of query (1, 1) is 2: a 3 by 3 matrix of 2s. The example was chosen to make the arithmetic small, and it has a cost, which the next section repairs.

Divide the scores by \(\sqrt{d_k} = \sqrt{2}\), which gives 1.414 everywhere, and apply the softmax to each row, which turns each row of scores into weights that are positive and sum to one. Equal scores give equal weights: each token attends to all three tokens with weight 1/3. The output for each token is its weights times the values, \(\frac{1}{3}(2, 1) + \frac{1}{3}(0, 1) + \frac{1}{3}(1, 1) = (1, 1)\). All three output rows are (1, 1). The complete formula is

\[ \mathrm{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \mathrm{softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d_k}}\right)\mathbf{V}, \]

and the example has walked through every symbol in it.

Making the lookup selective

Equal weights are what attention computes when the keys cannot tell the queries apart, and that is what the hand-set \(\mathbf{W}_K\) above did, since every row of \(\mathbf{X}\) has two 1s and the key matrix sums them. Change the key matrix so that the keys differ, say to one that copies the first two components of each token, and the keys become (1, 0), (0, 1), (1, 1). Now the first query (2, 0) scores 2 with the first key, 0 with the second, and 2 with the third. After scaling by \(\sqrt{2}\) the scores are 1.414, 0, 1.414, and the softmax gives weights of 0.45, 0.11, 0.45 (the exponentials are 4.11, 1, and 4.11, which sum to 9.22). The first token now attends mostly to itself and to the third token and little to the second. Its output is \(0.45 \times (2, 1) + 0.11 \times (0, 1) + 0.45 \times (1, 1) = (1.34, 1.00)\). The weights are what the heat maps in papers show: for each token, a row of numbers that say where it looked.

Why the square root

If the components of queries and keys are of size about 1, a dot product over \(d_k\) components has a size about \(\sqrt{d_k}\). For \(d_k = 64\), as in the original transformer, raw scores of plus or minus 8 would be typical, and a softmax over scores that far apart puts almost all its weight on the largest one: the weights become 0.999 and 0.001, the lookup becomes hard rather than soft, and the gradient through the softmax becomes nearly zero, so the model stops learning. Dividing by \(\sqrt{d_k}\) brings the scores back to size about 1 and keeps the softmax in the range where it is both selective and trainable. Multiply the scores of the selective example above by 10 instead and the weights become 0.50, 0.00, 0.50: the second token has been cut out entirely.

What is built on it

A transformer runs this computation several times in parallel with different learned matrices (multi-head attention, eight heads in the 2017 model), so that one head can look at the previous word, another at the subject of the sentence, and another at a matching bracket. It adds the output to the input (a residual connection), normalizes, passes the result through a small feed-forward network, and repeats the whole block six, twelve, or ninety-six times. Attention by itself has no idea of order, since the dot products do not care which row came first, so a positional encoding is added to the token vectors before the first block. And in a model that generates text, each token is only allowed to attend to the tokens before it, which is done by setting the scores of later tokens to minus infinity before the softmax, the causal mask.

The cost is the 3 by 3 score matrix, which for \(T\) tokens is \(T\) by \(T\). At 1,000 tokens that is a million scores per head per layer; at 100,000 tokens it is ten billion, and the memory to hold them in 16-bit floats is 20 gigabytes per head per layer. The quadratic cost is why context windows were short for years and why much of the engineering since 2020 has gone into computing attention without ever storing the whole matrix.

Unit 11 of the AI Masterclass works this example, computes the positional encoding for a four-dimensional model, counts the 65 million parameters of the original transformer block by block, and explains what the heads of a trained model learn. The article on tokens explains where the three rows of \(\mathbf{X}\) come from.

Learn how machines learn, properly

The AI Masterclass: weekly units, quizzes, a final exam, and a certificate.
About the program