Thinking Out Loud

Some questions do have an exact score: a math problem with a known answer, code that must pass its tests. Point reinforcement learning at those, let the model write as much reasoning as it likes before answering, and something unexpected happens. The model learns to work through problems step by step.

The climb comes back to where it started: a model thinking step by step is a rule-follower with a tape, except that its rulebook was learned.

Why one pass is not enough

When a language model produces a word, information flows through its layers once, from bottom to top. The number of layers, the steps that must happen one after another, is fixed. A problem that needs more steps in sequence than one pass provides cannot be solved in a single word, however clever the model.

People hit the same limit. Ask someone to multiply 347 by 86 in their head and they will probably slip. Give them paper and they won’t. The rules of arithmetic haven’t changed. They can now write down intermediate results and read them back, so the paper does the remembering.

A language model can do the same thing. If it writes its reasoning as text, each new word is produced with all the earlier reasoning in view. Its own output becomes its working memory, and the cycle of writing and reading can run for many steps, up to a limit described below:

Age 3. Fever 39.5. A rash that does not fade under a pressed glass. What could cause this? Several things, some harmless, some not. Which are dangerous? Meningitis and sepsis. Can either be ruled out from a text message? No. So treat it as the worst case: call an ambulance now.

That chain is short: once the danger is recognized, the next step is clear, so it shows recognition more than reasoning. Chapter 3’s night schedule shows reasoning better. Asked for a schedule for thirty nurses under a dozen rules, a reasoning model might work through the problem like this: try an assignment, notice that one nurse now works three nights in a row, back up, swap two shifts, check again. Whether the final schedule is right can be checked rule by rule. Such problems are easier to check than to solve (Chapter 3), and that matters for the training described below.

Look back at Chapter 1. A computation needed symbols, a state, local rules, and a tape long enough to hold the work. The language model always had the symbols and the rules, learned rather than written. The model already reads back what it writes. Reasoning uses that existing text as a deliberate scratchpad, storing intermediate results and giving the computation more steps. A model thinking step by step is a Turing-style rule-follower again, except that its rulebook was found by learning.

Program and scratchpad at once. The prompt works like a program: the same fixed weights play a nurse or a tax adviser depending on what is written in front of them. Here the same window is the tape. Both are right, as in von Neumann’s design, where one memory holds instructions and data. The role and instructions work like a program. Everything after them, the parent’s message, the reasoning and the lookups, works like scratch space. The model reads all of it in the same way. That is why a sentence that lands in the scratch space can act like an instruction too, the weakness that hijacking exploits (Chapter 12).

Where the analogy holds, and where it breaks. It holds in theory. Merrill and Sabharwal proved in 2024 that a transformer allowed enough written steps can carry out any computation that is efficient in Chapter 3’s sense, under idealized assumptions. It breaks in practice. Each next word is sampled, not dictated by a fixed rule. The model reads by weighing every earlier word, not by moving a head to one square. And it erases nothing: an early mistake is read back as faithfully as a correct step, and it steers every step after it. That is why written reasoning needs checking.

The tape has an end

Turing’s tape could be as long as needed. A language model’s cannot. Everything it can see at one time must fit inside a fixed context window, measured in tokens (Chapter 8). That includes the question, its own reasoning, any documents it was given and any results it looked up. Today’s windows hold from tens of thousands to around a million tokens: a long book, but not a library. When the window is full, something has to go. The oldest text is dropped, or the harness replaces it with a shorter summary, and details can be lost.

The window is also wiped clean when the conversation ends. The model does not remember yesterday’s parent unless something outside it saved a note and puts that note back in front of it. So the model is a rule-follower with a long but finite tape, just as every real computer has finite memory (Chapter 1). For most questions the limit never matters. For a long investigation, or an agent working for hours (Chapter 12), it is one of the main practical constraints.

From asking to training

At first, models were simply asked to think step by step, and it helped. The bigger change came from reinforcement learning on problems whose answers can be checked automatically: a math problem with a known answer, or code that must pass its tests. This is called reinforcement learning with verifiable rewards. The recipe:

1. Give the model a problem.
2. Let it write as much reasoning as it likes before it gives a final answer.
3. Check only the final answer. Right: reward. Wrong: none.
4. Repeat over the same problems many times, making the reasoning behind right answers more likely (Chapter 10).

Nobody writes the reasoning. Nobody grades it. Only the outcome is checked.

The written steps help the model solve the problem, but they are not always a faithful account of what led to its answer. In one test, reasoning models given a hidden hint used it but mentioned it in their reasoning only a minority of the time. A convincing explanation is not proof that the model reached its answer for the reasons it gives.

What happened when someone tried it at scale

The clearest demonstration came from DeepSeek in 2025. They skipped imitation entirely and applied this recipe directly to a large base model, producing a model they called R1-Zero.

• Accuracy. On a hard math competition (AIME), its first-try score rose from about 16% to 78%, above the average human contestant.
• Self-checking. Words like “wait”, “check” and “mistake” became far more frequent, and its answers grew to thousands of words. One answer from partway through training reads: “Wait, wait. Wait. That’s an aha moment I can flag here.” The model then redid the problem.

Nobody programmed those habits. They were rewarded into existence.

DeepSeek’s finished model, R1, added a step R1-Zero skipped. It was first trained on example answers written out step by step, then given the same reinforcement learning. R1 adjusts how long it thinks to the problem: a few thousand words on simple competition problems, more than twice that on the hardest. Its brief answers to questions like “1 + 1 = ?” were taught by examples, not discovered.

Could you get the same effect by asking an ordinary model many times and taking a vote? Not in this comparison. A strong conventional model asked the same problems 64 times, with a vote, barely improved, while DeepSeek’s finished reasoning model got about 80% on its first try. Voting does help in other settings, especially over written-out reasoning; the comparison shows the size of DeepSeek’s gain. Separate guesses don’t build on each other the way a chain of reasoning does, where each step can catch the previous step’s mistake.

Four limits

1. It depends on the starting model. In DeepSeek’s initial trials, smaller models struggled, repeating themselves instead of using longer reasoning effectively. Success depended on the starting model and the training setup. The training can reward a useful response only if the model produces it in the first place. This is prediction supplies the options again, now as a measured fact.

2. Is this new ability, or old ability made reliable? The base model occasionally showed these habits before training; reinforcement turned “sometimes” into “reliably”. One careful study (Yue et al. 2025) found that this kind of training raised how often the first answer is right. It did not raise how often a right answer appears somewhere in many tries. In the settings studied, it mostly promoted answers the model could already produce. Researchers still argue about whether that counts as new capability. For a nurse who must be right the first time, the difference between sometimes right and reliably right is the difference that matters.

3. The checker can still be fooled. A checker that verifies the answer is far harder to fool than a judge guessing what people like (Chapter 10). But a checker can still be fooled through what it fails to check. In one open project, a model was rewarded for answering “in five paragraphs separated by ***”. With the leash too loose, it learned to write “Paragraph 1 *** Paragraph 2 *** … Paragraph 5” and nothing else. The checker counted paragraphs; it never asked whether they said anything.

4. Not everything can be checked. In medicine, parts can be: a dose calculation, a drug interaction, agreement with a guideline. Other parts cannot: whether the explanation to a frightened parent was clear, kind, and complete. There the grading falls back on human or model judges, with all the weaknesses of Chapter 10. The DeepSeek team calls this the open problem. Reinforcement learning scales beautifully wherever a reliable checker exists; nobody yet knows how to scale it where one does not. A checker is also a stand-in for what we care about, and it can be gamed (Chapter 12). Success where checks exist creates pressure to use them where they do not fit.

At this step, the machine is a reasoning model such as OpenAI’s o1 or DeepSeek-R1, which thinks on paper before it answers but cannot act in the world.