Image: Willi Heidelbach (CC BY 2.5)What a token is and why your sentence costs more than mine
Ask a language model what the capital of France is and it never sees the word "capital". It sees a list of integers, each standing for a piece of text from a fixed vocabulary of tens of thousands of pieces. Common words are one piece; rare words are several; a word in Hindi may be a piece per letter. The pieces are called tokens, the vocabulary was built by an algorithm from 1994 that counted pairs, and the counting explains why some people's sentences cost more than others'.
Why not words
The obvious vocabulary is the words of the language. It fails twice. English has hundreds of thousands of words plus every name, every misspelling, and every new coinage, and a model with a word vocabulary would have to treat each one it had not seen as "unknown". And the model has to handle Python, French, chemical formulas, and emoji in the same text. The other obvious vocabulary is the characters, about 256 bytes. That never meets an unknown, but a sentence of 100 characters becomes 100 tokens, each carrying almost no meaning on its own, and the model's cost grows with the square of the number of tokens.
Byte pair encoding sits between the two. Start with the characters. Count every adjacent pair in the training text and merge the most frequent pair into a new token. Repeat until the vocabulary has the size you want. Frequent words end up as single tokens, rare ones as a few pieces, and nothing is ever unknown because the characters are still there underneath.
Four merges on five words
A corpus with five distinct words and their counts: "low" 5 times, "lower" 2, "newest" 6, "widest" 3, "lowest" 2. Write each word as its characters with an end-of-word marker, so that "low" is l o w </w>. Now count pairs across the corpus, weighting each by the word's count.
- Merge 1. The pair (e, s) occurs in newest, widest, and lowest: 6 + 3 + 2 = 11 times, the most frequent. Merge it into a token "es". The vocabulary gains one entry.
- Merge 2. The pair (es, t) now occurs 11 times. Merge into "est".
- Merge 3. The pair (est, </w>) occurs 11 times. Merge into "est</w>", a token that means "the suffix est at the end of a word".
- Merge 4. The pair (l, o) occurs in low, lower, and lowest: 5 + 2 + 2 = 9 times. Merge into "lo".
After four merges, "newest" is tokenized as n e w est</w>, four tokens where there were seven characters, and the algorithm has discovered, from counts alone, that English has a superlative suffix. A few more merges give "low</w>" as one token and "er</w>" as another. The merges are stored in order, and a new text is tokenized by applying them in the same order, so that "slowest", which was never in the corpus, becomes s lo w est</w>.
The example is small; the real procedure is the same. GPT-2 and GPT-3 used a vocabulary of 50,257 tokens learned from web text, with the merges applied to bytes rather than characters so that any string in any encoding can be represented. Newer models use around 100,000 to 200,000 tokens. The algorithm was published in 1994 by Philip Gage as a compression method and was brought into neural machine translation by Rico Sennrich, Barry Haddow, and Alexandra Birch in a 2016 paper whose example corpus this article borrows.
Why your sentence costs more
The merges are chosen by frequency in the training text, and the training text of the large models is mostly English. So English words become single tokens and other languages do not. "The weather is nice today" is about six tokens in English. The same sentence in German, with its compound words, takes somewhat more; in Hindi, Japanese, or Arabic, whose scripts occupy several bytes per character and whose words were rarely merged, it can take two to three times as many, and in some languages five times. Researchers at the University of Oxford measured the ratio across languages in 2023 and found that the same text could be up to about fifteen times longer in tokens for the least represented scripts than in English, with the exact numbers depending on the tokenizer.
Three consequences follow, and all three are felt by users. A model's context window is counted in tokens, so a Hindi speaker fits less text into it. Services that charge per token charge the Hindi speaker more for the same words. And the model's own competence suffers, because a word broken into five arbitrary pieces is harder to learn than a word that is one token. Numbers have the same problem: "2026" may be one token and "20261" two, split in a place that has nothing to do with arithmetic, which is one reason language models have been poor at multiplication. Later tokenizers split digits one at a time on purpose.
What a token is not
A token is not a word, a syllable, or a unit of meaning. It is a frequent string. The token for " the" (with its leading space) is different from the token for "the" at the start of a sentence, and "Token" and " token" are different tokens again. The model learns that these are related, which costs it some of its capacity. When a model is said to have been trained on 300 billion tokens, that is about 225 billion English words, since the usual ratio is about four tokens for three words, or about 200 million books of average length. When it is said to produce 50 tokens per second, that is about 40 words.
Tokenization is also the one part of a language model that is not learned by gradient descent. The merges are fixed before training by counting, and every later part of the model, the embeddings, the attention, the output distribution, works in terms of them. A different vocabulary would be a different model, and a mistake in the tokenizer, such as the token in GPT-2's vocabulary that was learned from a Reddit user's name and caused strange outputs years later, is baked in for the life of the model.
Unit 13 of the AI Masterclass works byte pair encoding on this corpus, records the token counts of one sentence in several languages with a named tokenizer, and uses the counts to compute the cost of a training run from the number of tokens. Unit 10 introduces tokens with embeddings, and the article on attention shows what the model does with them.


