Embeddings and Sequence Models
A word is a symbol and a sentence has no fixed length, so neither fits the networks of the earlier units. This unit turns words into vectors that carry meaning, learned from nothing except which words appear near which, and builds the recurrent networks that read a sentence one word at a time. It shows by hand why such a network forgets after a few steps, how the gates of the LSTM fix that, how an encoder and a decoder translate, and what the perplexity of a language model measures.
Every input of Units 1 to 9 was a vector of fixed length, such as a flower described by four measurements or an image of 784 pixels. A sentence is neither of these things, because its words are symbols rather than numbers and because sentences come in every length. This unit solves both problems in turn. First it turns a word into a vector that carries meaning, learned from nothing except which words appear near which, and then it builds networks that read a sentence one word at a time and carry what they have read along with them. The two ideas, the embedding and the recurrent network, were the machinery of translation, speech recognition, and text prediction until 2017, and the first of them is still inside every language model built since.
1 Words as numbers
A network needs numbers, and the obvious way to give it a word is to number the vocabulary. With a vocabulary of \(V\) words, the word with number \(j\) becomes the one-hot vector of length \(V\) that has a 1 in position \(j\) and zeros everywhere else. Figure 1 shows three such vectors for a vocabulary of eight words. A real vocabulary has 50,000 words or more, so every word becomes a vector of 50,000 components, of which one is used.
The representation adds no knowledge about the words, and that is its defect. The dot product of Unit 1 measured how alike two vectors are, and the dot product of two different one-hot vectors is always zero. In this representation “cat” is exactly as far from “dog” as from “carburettor”, and whatever a network learns about cats it must learn again, from scratch, for dogs. The question is where a better set of numbers could come from, since nobody can sit down and write a few hundred numbers that describe a cat.
The answer came from linguistics, a decade before anyone could compute with it. Zellig Harris (1909-1992) argued in a 1954 paper in the journal Word that the meaning of a word is reflected in the set of contexts in which it occurs, and that two words which occur in similar contexts have similar meanings. J. R. Firth (1890-1960) put the same idea into a sentence in 1957 that the field has quoted ever since: “You shall know a word by the company it keeps.”
Co-occurrence can be counted, and counts are numbers. The crude version of the idea counts, for every word, how often each other word appears within a few positions of it, and the \(V\) counts of a word form its vector. This gives “cat” and “dog” similar vectors, because both co-occur with “fur”, “vet”, and “sleeps”, but the vectors have \(V\) components and are mostly zeros. The modern version compresses them. An embedding of a word is a vector of \(d\) components that a learning algorithm has adjusted so that words with similar contexts receive similar vectors. For word vectors of this kind \(d\) lies between about 50 and a few hundred, and in the largest language models of today it is several thousand. The embeddings of all \(V\) words form the embedding matrix \(\mathbf{E}\) with \(V\) rows and \(d\) columns, and the matrix-vector product of Unit 1 between the transpose \(\mathbf{E}^{\top}\) (the same table with rows and columns swapped, as in Unit 6) and a one-hot vector picks out exactly one row of \(\mathbf{E}\), the embedding of that word. In code the multiplication is skipped and the row is looked up directly. With \(V = 50{,}000\) and \(d = 300\) the matrix holds 15 million numbers, and such a matrix is the first layer of almost every language model.
1.1 Measuring similarity
Two embeddings are compared by the angle between them rather than by the dot product alone, because the dot product also grows with the lengths of the vectors, and in practice the length of a word vector reflects how often the word was seen rather than what it means. The dot product of two vectors equals the product of their lengths and the cosine of the angle between them, which is the geometry behind the projection drawing of Unit 1, and dividing the lengths out gives the standard measure.