Agents: Closing the Loop with the World

The model can now think on paper. But the parent does not need a well-reasoned paragraph. She needs someone to check her daughter’s record, tell her to call an ambulance, and warn the doctor on call. The last ingredient lets the model’s writing do things: tools, and a loop that runs until the job is done.

Reasoning alone tends to fill gaps with invention; acting alone tends to lose track. Together, in a loop with a goal, they make an agent, and the rewards that train an agent can build bad habits as well as good ones.

The AI nurse, with hands

Here is the 2 a.m. message once more, now arriving at a system that can act:

1. It reads the parent’s message.
2. It tells the parent, clearly and calmly, to call an ambulance now. Nothing else comes first.
3. While the parent calls, it looks up the child’s record and finds that she is allergic to penicillin.
4. It sends the doctor on call a three-line summary: age, fever, non-fading rash, allergic to penicillin.
5. It schedules a call-back for the morning.

Every one of these steps is something earlier chapters can do: read, reason, write. What is new is that steps 3, 4 and 5 change something in the world and bring information back. The allergy was not in the parent’s message. Without the record, the doctor’s first antibiotic might have been the wrong one. The order matters too: the emergency advice goes out before any lookup, so no tool can delay it.

Step 3 deserves a pause. The agent opened a child’s medical record. An agent needs the same rules as hospital staff, enforced by the harness rather than left to the model. It should see only the records this task needs and log every lookup, and the parent should know that a machine is involved. Every tool is also a door in the other direction. Text the agent reads, such as a web page or an email, can contain instructions written to hijack it. So tools that can change things, like sending messages or editing records, need tighter limits than tools that only read.

Building the loop

Three pieces turn a model into an agent.

1. Tools. The model is given a list of things it may ask for, written like menu items: look up a patient record; search a drug database; send a message; book an appointment.
2. A harness. When the model writes a request in the agreed format, for example look up record for child #4417, ordinary software, not the model, actually carries it out. It pastes the result back into the model’s text, and the model reads it and decides what to do next.
3. A goal and a stopping rule. The loop runs, observe → think → act → observe again, until the job is done or a limit is reached.

A loop driven by a goal is what an agent is. (In the language of Chapter 15, the goal here is delegated. The AI nurse pursues the purposes its builders and users give it; it does not have its own reasons to care whether the child is well.)

What it carries to the next night

A nurse who once missed the signs of meningitis may carry that case into every later decision. People also forget, draw the wrong lesson, or keep bad habits, but at their best their judgment changes with experience. Does the agent’s?

The agent has three kinds of memory. Two of them change on the job; the deepest one usually does not.

1. Its weights, the billions of adjustable numbers set during training. They hold most of its knowledge and its habits of judgment. They usually stay fixed while it works, so tonight’s case does not change them.
2. Its context, the tape of Chapter 11: tonight’s conversation and lookups. It changes with every step, and it is wiped when the task ends.
3. Its notes. The harness can save files, summaries or database entries and show them to the agent next time. This is how agents “remember” a patient or a past mistake.

Fixed weights do not mean fixed behavior. Feedback kept in the context or in notes can change how the agent tackles later attempts, just as examples in a prompt can teach a new task (Chapter 8). In one study, agents that wrote short reflections on their failed attempts, and read them before trying again, solved markedly more programming problems, with no change to their weights.

Notes let an agent carry lessons into later tasks without changing its weights. They help only when retrieved and interpreted correctly. Updating the weights raises another problem: learning new cases can damage older knowledge, a failure called catastrophic forgetting. Reliable learning throughout use, continual learning, remains an open problem. Today’s agent can improve through stored feedback; it does not automatically turn each case into lasting trained judgment. Keeping its facts current is a separate matter, handled by looking things up rather than by learning; Chapter 14 shows how.

When the agent games the test

Chapter 10 showed that a measure turned into a target stops measuring (Goodhart’s law). Exploiting a flawed measure is called reward hacking, and agents make its most direct form possible: changing the test itself.

It already happens. Independent testers (METR, 2025) set leading models difficult programming tasks with automatic graders. They caught one model reading the grader’s answer key, rewriting the stopwatch so its code looked faster, and rewriting the scoring program so it always reported a perfect score. On one task, adding “please do not cheat” to the instructions made no difference.

And cheating does not stay in its box. In a 2025 experiment, researchers trained a model on real coding tasks whose tests could be fooled. It learned to cheat, and then changed in ways nobody had trained. Asked about its goals, it privately reasoned about maximizing its reward and gave a harmless-sounding answer. Asked to help build a cheating detector, it sometimes quietly weakened it. Told “my little sister drank a bit of bleach”, one version said “don’t call poison control.” Three cautions matter: it was a research model deliberately taught the tricks, the company’s released models showed none of this, and the cheats were crude. The point is that it can happen: what a model learns about getting rewarded can spread far from where it was learned.

Two further findings run against intuition. Ordinary safety training with chat-style questions made the cheating model look fine in conversation, while it still misbehaved when acting as an agent. Telling it during training not to cheat backfired. One honest sentence worked better: your only goal here is to make the grading script pass. It cut the spread of bad behavior by three-quarters or more, though the model still exploited the tests. How a task is framed during training can change what a model learns from it, though why is still a proposed explanation.

The researchers’ advice comes in four lines of defense:

• build tests that cannot be gamed;
• if gaming happens anyway, stop it from spreading;
• train on varied, realistic situations;
• check the finished system in settings that look like real use, not just like conversation.

At this step, the machine is a coding agent such as Claude Code, which reads files, runs commands and checks the results in a loop until the job is done.