The walls say what no rulebook can do. But the nurse’s knowledge does not come near them: deciding that a feverish child with a non-fading rash needs an ambulance is neither undecidable nor astronomically slow. The real problem is more ordinary. Somebody has to write the rulebook. In the 1970s, researchers tried to do exactly that for medicine.

We know a great deal of medicine. Writing it down as rules works, within limits, and the limits are exactly where real conversations live.

A program that knew about meningitis

In the early 1970s, at Stanford University, a medical student and computer scientist named Edward Shortliffe built a program called MYCIN for his doctoral thesis. Its job was narrow and serious: help doctors choose antibiotics for severe bacterial infections of the blood. Later it took on meningitis, the very illness a nurse would fear in our feverish child.

MYCIN’s knowledge was a list of about 600 rules, written with the help of infection specialists. Each rule had the form if these facts hold, then conclude that, with this much confidence. A rule might say, in effect: if the infection was picked up in hospital, and the patient has had a certain kind of procedure, then there is suggestive evidence that the bacterium is of a certain type. The program asked the doctor questions, chained rules together to reach conclusions, and recommended drugs with doses adjusted for the patient’s weight. It could also explain itself: asked why?, it showed the rules it was following.

Did it work? In a small blinded study of ten meningitis cases, published in 1979, infection specialists judged MYCIN’s recommendations at least as acceptable as those of the faculty physicians. Within a narrow field, hand-written rules could match expert performance.

MYCIN never treated patients, mostly for practical reasons. It ran on a large shared computer, and every answer had to be typed in by hand. And nobody knew who would be responsible if its advice went wrong. Its ideas, though, launched a whole industry. Through the 1980s, companies built expert systems for everything from configuring computers to approving loans.

Capturing the nurse in rules

The idea behind expert systems is simple and attractive. An expert knows things. Write them down as rules:

IF fever AND rash AND the rash does not fade when pressed THEN advise emergency care.

A universal machine (Chapter 2) can follow any rulebook, so it will follow this one. The rules can be read, checked and corrected by other experts. Every conclusion comes with an explanation: because rules 12 and 47 applied. Much of medicine already works this way. Public guidance tells parents to call an emergency number if a feverish child has a rash that does not fade when a glass is pressed against it. That is a rule, and a good one.

Why not keep going until the whole nurse is written down? The expert-system era found two problems, and neither was a lack of effort.

Problem 1: brittleness

Here is the parent’s message again:

“…little purple spots on her legs, and they don’t go away when I press a glass on them.”

The rule says rash does not fade when pressed. The parent did not use the word “rash”. She did not say “fade”. This hand-written matcher recognizes only the wording its authors have provided for. Unless someone wrote a rule connecting “little purple spots” to “rash” and “don’t go away” to “does not fade”, the emergency rule never fires.

So you add those rules. Then another parent writes “tiny dark red dots, the glass trick doesn’t make them disappear”, and another writes in a second language, and another misspells “glass”. Then you discover that “spots that go when I press” means the opposite and must not trigger the rule. Every new rule interacts with the old ones. The rulebook grows, and the cases it misses never run out.

This is called brittleness. A rule-based system works well inside the world its authors imagined and fails suddenly, without warning, just outside it. It does not get slightly worse. It falls off a cliff.

Problem 2: the knowledge bottleneck

The second problem is deeper. To write the rules, you must first get them out of the experts’ heads. Builders of expert systems found this was the slowest and most expensive part of the whole job. They called it the knowledge acquisition bottleneck.

Part of the reason is that experts know more than they can say. An experienced children’s nurse can often tell that a child “looks wrong” across a room: something about the color, the stillness, the way the child does or does not respond. Ask her for the rule and she will give you a list, and the list will not be everything she uses. The philosopher Michael Polanyi put it in one line: we can know more than we can tell. Knowledge of this kind, which experts use but cannot fully state, is called tacit knowledge.

Tacit knowledge is not rare or mysterious. You use it to recognize a friend’s face, to understand a sarcastic remark, or to know that a sentence sounds wrong. You could not write the rules for any of these, yet you follow them every day.

At this step, the machine is MYCIN, the 1970s Stanford program that chose antibiotics as well as specialists did, by following rules that experts wrote by hand.