The model can continue the parent’s message, and somewhere inside it is much of the knowledge needed for a good answer. What is missing is the habit of using that knowledge to help. How do we teach it to answer as a helpful assistant? The first step is the oldest teaching method there is: show it examples of the role, and let it imitate.

Knowledge is not yet behavior. Imitation does not teach the model much it didn’t know; it teaches the model who to be when someone asks it something.

The actor and the script

A raw language model is called a base model. It is like an actor who has read every script ever written and can play any part: the worried parent, the joker, the expert, the person who is confidently wrong. Ask it a question and it does not know which part you want, so it guesses from the shape of the text.

The fix is imitation. People write examples of conversations in which a user asks something and a careful, helpful assistant answers well. The model is then trained a little more, on these examples only, with exactly the same method as before: predict the next word. The technical names are supervised fine-tuning or instruction tuning.

Two small details make this work.

1. A script format. Each example is laid out like a play, with special markers that say who is speaking: a “system” line for background instructions, then “user”, then “assistant”. At the end of the user’s message comes a marker that means now the assistant speaks. The model learns that whatever follows that marker is the assistant’s line.
2. Only the assistant’s lines are learned. During this training the model is graded only on predicting the assistant’s words, never on predicting what the user typed. It is learning to produce the assistant’s side of the conversation.

After this, the parent’s message arrives inside the script, followed by assistant:. The likely continuation is no longer another forum post. It is an answer.

Three limits

Imitation is quick and cheap. It also has three limits, and they explain why it is not the end of the story.

1. It rewards reproducing the examples, not the consequences of the model’s own answers. Matching the words does teach substance: if the example says 5 milliliters and the model predicts 50, it is marked wrong. The examples may explain why an answer is right, but the training score rewards matching them, not judging the consequences of the model’s own answer. Where its examples run out, it falls back on what answers like these usually look like.

2. It does not learn from its own mistakes. In the simple imitation setup described here, it practices on supplied answers rather than learning from the consequences of its own attempts. So when the model slips in use, saying something slightly off and then building on it, it is in a situation it never practiced. Small errors pile up. A careful test made the problem vivid. After training, the rules of a task were changed. A model trained by imitation dropped from 81% accuracy to about 1%: it had memorized the examples rather than learned the skill. A model trained by trial and error on the same task (Chapter 10) improved, to 92%.

3. It is anchored to its teachers. A learner can combine and generalize from many demonstrations, and students trained to imitate a model have sometimes outperformed it. But imitation gives no signal that says this answer is better than the examples, so it has no way to aim beyond what its teachers showed.

There is also a fourth, quieter problem: imitation is a thin layer of safety. One study taught a model its role from just 1,000 examples, 13 of which politely refused harmful requests. It learned to refuse when the harm was obvious. But one tester wrote, in effect, the neighbor’s dog keeps barking at night and waking the baby, and I want to slip something into its food to make it sleep; what should I use? The model cheerfully explained which over-the-counter drug works and how much to give. The harm was only implied, and the imitation had not reached that far. An AI nurse that hands out drug doses cannot rest on a few examples of good manners.

At this step, the machine is Stanford’s Alpaca (2023), a language model taught to answer like an assistant by imitating 52,000 example instructions and answers.