Seeing needed labels: a million photos tagged with their category, a hundred thousand X-rays labeled from radiologists’ reports. Hearing needed transcripts. Labels are expensive and, as Chapter 7 showed, can carry hidden shortcuts. Is there a way to learn without anyone labeling anything? For language there is, and it changes everything.

Predicting the next word looks like a trivial task. Doing it well requires knowing a great deal about the world.

Fill in the blank

Try to predict the missing word:

The child has a high fever and a rash that does not fade when you press a glass on it. The parents should call an ___

You probably said “ambulance”. To get it, you needed grammar (“an” before a vowel sound), and also medicine: what a non-fading rash with fever can mean, and what people should do about it. Now try:

The capital of France is ___ / If 347 × 86 = 29,842, then 347 × 87 = ___ / “I’m fine,” she said, slamming the door. She was not ___

Each blank needs something different: a fact, a calculation, an understanding of people. A system that could fill in every blank in everything people have written, as well as possible, would have to know a great deal about what that writing is about.

This idea drives this chapter’s machine, and it also solves the problem Chapter 7 ended on. Nobody has to label anything. Every sentence is its own test: hide the next word, ask for a prediction, then reveal the answer. All the text people have ever put online becomes training data. This is called self-supervised learning.

A hundred years of predicting words

Counting. The oldest way to predict text is to count: look at the last two or three words, and guess whichever word most often followed them in a large collection. These n-gram models, which go back to Andrey Markov’s counts of letters in a Pushkin poem in 1913, served for decades in spell-checkers and early speech recognition. But most longer contexts never appear in any collection, so the model has nothing to say about them.

Words as points. In 2003 Yoshua Bengio and colleagues gave every word a list of numbers, a point in a space as in Chapter 7, learned along with the prediction. Words used in similar ways drift near each other, so a sentence never seen before still gets a sensible prediction. A later version, word2vec (2013), showed that directions in the space carry meaning: “king” minus “man” plus “woman” lands near “queen”. Nobody programmed this.

Attention. To predict the next word you may need something from far back: the patient’s name three paragraphs ago, or the negation at the start of the sentence. Networks that read one word at a time tended to forget. Attention (2014) lets the network look back over all the earlier words and decide, by learned similarity, which matter for the word it is about to produce. For “she was not ___”, it can pick out “I’m fine” and “slamming the door”.

The transformer. In 2017 a team at Google built a design from attention and almost nothing else: the transformer. Reading a whole passage at once lets it train on parallel hardware. It adds each word’s position to its list of numbers. This keeps “the dog bit the child” apart from “the child bit the dog”, closing the gap Chapter 5 left open. Inside, it is built from Chapter 5’s parts: in each layer, attention mixes information between positions in the text, and a small multiply-add-bend network transforms it at each position. Stack dozens of these layers and you have the model.

Scale. From 2018, OpenAI’s transformers, trained on plain next-word prediction, simply got bigger: from about 117 million adjustable numbers to 175 billion in 2020. The 2020 model showed something unexpected: it could do a new task from a few examples written into its prompt, with no further training. Show it three English–French word pairs and a fourth English word, and it continues with the French.

Why prediction teaches so much

Chapter 6 said that learning is compression: finding a short description that generates the examples. Next-word prediction is compression in its purest form. The training score is literally the number of bits the model would need to encode the text. A model can lower its score on the training text partly by memorizing passages. But to need fewer bits on new text, it has to capture structure that carries beyond the examples: grammar, then facts, then the patterns of argument, explanation and reasoning that people use when they write.

So a network that predicts human writing well has, in some sense, compressed a large part of what humans know and how they think, as recorded in text. The result is called a base model.

Two cautions belong here. First, it has captured patterns in text about the world. Whether that amounts to knowing the world is Chapter 14’s question. Second, predicting well on average does not mean being right on every fact. An arbitrary fact, such as someone’s birthday, cannot be inferred from general patterns, so where the text says it rarely, the model may fill the gap with something plausible.

How it picks the next word

Two practical details matter for everything that follows.

Tokens. The model does not actually read whole words. Text is cut into tokens: common words stay whole, and rarer ones are split into pieces, so “unbelievable” might become un, believ and able. On average a token is about three-quarters of an English word. A model’s vocabulary is a fixed list of tens of thousands to a few hundred thousand such pieces, enough to spell anything in any language. From here on, “predicting the next word” really means predicting the next token.

Probabilities, then a draw. The model never outputs a single word. For every token in its vocabulary it outputs a probability. After “The parents should call an ___”, “ambulance” gets most of the probability, “emergency” a little, and “elephant” almost none. To write, the model draws one token at random according to those probabilities, adds it to the text, and does the whole thing again for the next token. A setting called temperature controls how adventurous the draw is. At low temperature it almost always takes the top choice; at high temperature it spreads its choices wider.

This small detail has large consequences. Ask the same question twice and you may get two different answers. Where the model has learned something well, most of the probability sits on one answer, and repeated tries agree, though consistent answers can still be consistently wrong (Chapter 16). Where its knowledge is thin, the probability is spread out, and repeated tries disagree. Several later chapters rely on this.

The base model is not trained to answer

Now give the base model the parent’s message:

“My daughter is 3. She’s had a fever all evening, 39.5. Now she has little purple spots on her legs, and they don’t go away when I press a glass on them. Should I wait until morning?”

It may not answer. It may continue the document. To a base model, this message looks like the start of a forum post. A likely continuation is another post: “Same thing happened with my son last year, turned out to be…” Or more questions, as if for a help page. This is real behavior. Asked “What is the capital of France?”, a base model may carry on with “What is the capital of Germany? What is the capital of Italy?”. Given “The president of the United States in 2006 was”, a very large base model completes the sentence correctly and then runs on into unrelated scraps of web pages.

Nothing is broken. It was trained to continue text, and it does.

At this step, the machine is GPT-3 as released in 2020, which could continue any text you gave it but was not trained to answer you as an assistant.