“just like one of the family.”

Although the The Jetsons correctly predicted cell phones and smartwatches, we still don’t have anything like Rosey. As of this book going to print, even Rosey’s most basic behaviors are still out of reach. It is no secret that the first company to build a robot that can simply load a dishwasher will immediately have a bestselling product. All attempts to do this have failed. It isn’t fundamentally a mechanical problem; it’s an intellectual one—the ability to identify objects in a sink, pick them up appropriately, and load them without breaking anything has proven far more difficult than previously thought.

Of course, even though we do not yet have Rosey, the progress in the field of artificial intelligence (AI) since 1962 has been remarkable. AI can now beat the best humans in the world at numerous games of skill, including chess and Go. AI can recognize tumors in radiology images as well as human radiologists. AI is on the cusp of autonomously driving cars. And as of the last few years, new advancements in large language models are enabling products like ChatGPT, which launched in fall 2022, to compose poetry, translate between languages at will, and even write code. To the chagrin of every high school teacher on planet Earth, ChatGPT can instantly compose a remarkably well written and original essay on almost any topic that an intrepid student might ask of it. ChatGPT can even pass the bar exam, scoring better than 90 percent of lawyers.

Across this long arrow of AI achievements, it has always been hard to tell how close we are to creating human-level intelligence. After the early successes of problem solving algorithms in the 1960s, the AI pioneer Marvin Minsky famously proclaimed that “from three to eight years we will have a machine with the general intelligence of an average human being.” It did not happen. After the successes of expert systems in the 1980s, BusinessWeek proclaimed “AI: it’s here.” Progress stalled shortly thereafter. And now with advancements in large language models, many researchers have again proclaimed that the “game is over” because we are

“on the verge of achieving human-level AI.” Which is it: Are we finally on the cusp of creating human-like artificial intelligence like Rosey, or are large language models like ChatGPT just the most recent achievement in a long journey that will stretch on for decades to come?

Along this journey, as AI keeps getting smarter, it is becoming harder to measure our progress toward this goal. If an AI system outperforms humans on a task, does it mean that the AI system has captured how humans solve the task? Does a calculator—capable of crunching numbers faster than a human—actually understand math? Does ChatGPT—scoring better on the bar exam than most lawyers—actually understand the law? How can we tell the difference, and in what circumstances, if any, does the difference even matter?

In 2021, over a year before the release of ChatGPT—the chatbot that is now rapidly proliferating throughout every nook and cranny of society—I was using its precursor, a large language model called GPT-3. GPT-3 was trained on large quantities of text (large as in the entire internet), and then used this corpus to try to pattern match the most likely response to a prompt. When asked, “What are two reasons that a dog might be in a bad mood?” it responded, “Two reasons a dog might be in a bad mood are if it is hungry or if it is hot.” Something about the new architecture of these systems enabled them to answer questions with what at least seemed like a remarkable degree of intelligence. These models were able to generalize facts they had read about (like the Wikipedia pages about dogs and other pages about causes of bad moods) to new questions they had never seen. In 2021, I was exploring possible applications of these new language models—could they be used to provide new support systems for mental health, or more seamless customer service, or more democratized access to medical information?

The more I interacted with GPT-3, the more mesmerized I became by both its successes and mistakes. In some ways it was brilliant, in other ways it was oddly dumb. Ask GPT-3 to write an essay about eighteenth-century potato farming and its relationship to globalization, and you will get a surprisingly coherent essay. Ask it a commonsense question about what someone might see in a basement, and it answers nonsensically.

over one billion connections.

But the sheer number of connections is only one aspect of what makes the brain complex; even if we mapped the wiring of each neuron we would still be far from understanding how the brain works. Unlike the electrical connections in your computer, where wires all communicate using the same signal—electrons—across each of these neural connections, hundreds of different chemicals are passed, each with completely different effects. The simple fact that two neurons connect to each other tells us little about what they are communicating. And worst of all, these connections themselves are in a constant state of change, with some neurons branching out and forming new connections, while others are retracting and removing old ones. Altogether, this makes reverse engineering how the brain works an ungodly task.

Studying the brain is both tantalizing and infuriating. One inch behind your eyes is the most awe-inspiring marvel of the universe. It houses the secrets to the nature of intelligence, to building humanlike artificial intelligence, to why we humans think and behave the way we do. It is right there, reconstructed millions of times per year with every newly born human. We can touch it, hold it, dissect it, we are literally made of it, and yet its secrets remain out of reach, hidden in plain sight.

If we want to reverse-engineer how the brain works, if we want to build Rosey, if we want to uncover the hidden nature of human intelligence, perhaps the human brain is not nature’s best clue. While the most intuitive place to look to understand the human brain is, naturally, inside the human brain itself, counterintuitively, this may be the last place to look. The best place to start may be in dusty fossils deep in the Earth’s crust, in microscopic genes tucked away inside cells throughout the animal kingdom, and in the brains of the many other animals that populate our planet.

In other words, the answer might not be in the present, but in the hidden remnants of a long time past.

The Missing Museum of Brains

—GEOFFREY HINTON (PROFESSOR AT UNIVERSITY OF TORONTO, CONSIDERED ONE OF THE “GODFATHERS OF AI”)

Humans fly spaceships, split atoms, and edit genes. No other animal has even invented the wheel.

Because of humanity’s larger résumé of inventions, you might think that we would have little to learn from the brains of other animals. You might think that the human brain would be entirely unique and nothing like the brains of other animals, that some special brain structure would be the secret to our cleverness. But this is not what we see.

What is most striking when we examine the brains of other animals is how remarkably similar their brains are to our own. The difference between our brain and a chimpanzee’s brain, besides size, is barely anything. The difference between our brain and a rat’s brain is only a handful of brain modifications. The brain of a fish has almost all the same structures as our brain.

These similarities in brains across the animal kingdom mean something important. They are clues. Clues about the nature of intelligence. Clues about ourselves. Clues about our past.

Although today brains are complex, they were not always so. The brain emerged from the unthinking chaotic process of evolution; small random variations in traits were selected for or pruned away depending on whether they supported the further reproduction of the life-form.

In evolution, systems start simple, and complexity emerges only over time.

motor, and other functions.”

The problem is that MacLean’s Triune Brain Hypothesis has been largely discredited—not because it is inexact (all frameworks are inexact), but because it leads to the wrong conclusions about how the brain evolved

and how it works. The implied brain anatomy is wrong; the brains of reptiles are not only made up of the structures MacLean referred to as the “reptile brain”; reptiles also have their own version of a limbic system. The functional divisions proved wrong; survival instincts, emotions, and cognition do not delineate cleanly—they emerge from diverse networks of systems spanning all three of these supposed layers. And the implied evolutionary story turned out to be wrong. You do not have a reptile brain in your head; evolution did not work by simply layering one system on top of another without any modifications to the existing systems.

But even if MacLean’s triune brain had turned out to be closer to the truth, its biggest problem is that its functional divisions aren’t particularly useful for our purposes. If our goal is to reverse-engineer the human brain to understand the nature of intelligence, MacLean’s three systems are too broad and the functions attributed to them too vague to provide us with even a point at which to start.

We need to ground our understanding of how the brain works and how it evolved in our understanding of how intelligence works—for which we must look to the field of artificial intelligence. The relationship between AI and the brain goes both ways; while the brain can surely teach us much about how to create artificial humanlike intelligence, AI can also teach us about the brain. If we think some part of the brain uses some specific algorithm but that algorithm doesn’t work when we implement it in machines, this gives us evidence that the brain might not work this way. Conversely, if we find an algorithm that works well in AI systems, and we find parallels between the properties of these algorithms and properties of animal brains, this gives us some evidence that the brain might indeed work this way.

The physicist Richard Feynman left the following on a blackboard shortly before his death: “What I cannot create, I do not understand.” The brain is our guiding inspiration for how to build AI, and AI is our litmus test for how well we understand the brain.

We need a new evolutionary story of the brain, one grounded not only in a modern understanding of how brain anatomy changed over time, but also in a modern understanding of intelligence itself.

The Five Breakthroughs

—YANN LECUN, HEAD OF AI AT META

We have a lot of evolutionary history to cover—four billion years. Instead of chronicling each minor adjustment, we will be chronicling the major evolutionary breakthroughs. In fact, as an initial approximation—a first template of this story—the entirety of the human brain’s evolution can be reasonably summarized as the culmination of only five breakthroughs, starting from the very first brains and going all the way to human brains.

These five breakthroughs are the organizing map to our book, and they make up our itinerary for our adventure back in time. Each breakthrough emerged from new sets of brain modifications and equipped animals with a new portfolio of intellectual abilities. This book is divided into five parts, one for each breakthrough. In each section, I will describe why these abilities evolved, how they worked, and how they still manifest in human brains today.

Each subsequent breakthrough was built on the foundation of those that came before and provided the foundation for those that would follow. Past innovations enabled future innovations. It is through this ordered set of modifications that the evolutionary story of the brain helps us make sense of the complexity that eventually emerged.

But this story cannot be faithfully retold by considering only the biology of our ancestors’ brains. These breakthroughs always emerged from periods when our ancestors faced extreme situations or got caught in powerful feedback loops. It was these pressures that led to rapid reconfigurations of brains. We cannot understand the breakthroughs in brain evolution without also understanding the trials and triumphs of our ancestors: the predators they outwitted, the environmental calamities they endured, and the desperate niches they turned to for survival.

And crucially, we will ground these breakthroughs in what is currently known in the field of AI, for many of these breakthroughs in biological intelligence have parallels to what we have learned in artificial intelligence. Some of these breakthroughs represent intellectual tricks we understand well in AI, while other tricks still lay beyond our understanding. And in this way, perhaps the evolutionary story of the brain can shed light on what breakthroughs we may have missed in the development of artificial humanlike intelligence. Perhaps it will reveal some of nature’s hidden clues.

Me

I wish I could tell you that I wrote this book because I have spent my whole life pondering the evolution of the brain and trying to build intelligent robots. But I am not a neuroscientist or a roboticist or even a scientist. I wrote this book because I wanted to read this book.

I came to the perplexing discrepancy between human and artificial intelligence by trying to apply AI systems to real-world problems. I spent the bulk of my career at a company I cofounded named Bluecore; we built software and AI systems to help some of the largest brands in the world personalize their marketing. Our software helped predict what consumers would buy before they knew what they wanted. We were merely one tiny part in a sea of countless companies beginning to use the new advances in AI systems. But all these many projects, both big and small, were shaped by the same perplexing questions.

When commercializing AI systems, there is eventually a series of meetings between business teams and machine learning teams. The business teams look for applications of new AI systems that would be valuable, while only the machine learning teams understand what applications would be feasible. These meetings often reveal our mistaken intuitions about how much we understand about intelligence. Businesspeople probe for applications of AI systems that seem straightforward to them. But frequently, these tasks seem straightforward only because they are straightforward for our brains. Machine learning people then patiently explain to the business team why the idea that seems simple is, in fact, astronomically difficult. And these debates go back and forth with every new project. It was from these explorations into how far we could stretch modern AI systems and the surprising places where they fall short that I developed my original curiosity about the brain.

Of course, I am also a human and I, like you, have a human brain. So it was easy for me to become fascinated with the organ that defines so much of the human experience. The brain offers answers not only about the nature of intelligence, but also why we behave the way we do. Why do we frequently make irrational and self-defeating choices? Why does our species have such a long recurring history of both inspiring selflessness and unfathomable cruelty?

My personal project began with merely trying to read books to answer my own questions. This eventually escalated to lengthy email correspondences with neuroscientists who were generous enough to indulge the curiosities of an outsider. This research and these correspondences eventually led me to publish several research papers, which all culminated in the decision to take time off work to turn these brewing ideas into a book.

Throughout this process, the deeper I went, the more I became convinced that there was a worthwhile synthesis to be contributed, one that could provide an accessible introduction to how the brain works, why it works the way it does, and how it overlaps and differs from modern AI systems; one that could bring various ideas across neuroscience and AI together under an umbrella of a single story.

A Brief History of Intelligence is a synthesis of the work of many others. At its heart, it is merely an attempt to put together the pieces that were already there. I have done my best to give due credit throughout the book, always aiming to celebrate those scientists who did the actual research. Anywhere I have failed to do so is unintentional. Admittedly, I couldn’t resist sprinkling in a few speculations of my own, but I will aim to be clear when I step into such territory.

It is perhaps fitting that the origin of this book, like the origin of the brain itself, came not from prior planning but from a chaotic process of false starts and wrong turns, from chance, iteration, and lucky circumstance.

A Final Point (About Ladders and Chauvinism)

I have one final point to make before we begin our journey back in time. There is a misinterpretation that will loom dangerously between the lines of this entire story.

This book will draw many comparisons between the abilities of humans and those of other animals alive today, but this is always done by picking specifically those animals that are believed to be most similar to our ancestors. This entire book—the five-breakthroughs framework itself—is solely the story of the human lineage, the story of how our brains came to be; one could just as easily construct a story of how the octopus or honeybee brain came to be, and it would have its own twists and turns and its own breakthroughs.

Just because our brains wield more intellectual abilities than those of our ancestors does not mean that the modern human brain is strictly intellectually superior to those of other modern animals.

Evolution independently converges on common solutions all the time. The innovation of wings independently evolved in insects, bats, and birds; the common ancestor of these creatures did not have wings. Eyes are also believed to have independently evolved many times. Thus, when I argue that an intellectual ability, such as episodic memory, evolved in early mammals, this does not mean that today only mammals have episodic memory. Like with wings and eyes, other lineages of life may have independently evolved episodic memory. Indeed, many of the intellectual faculties that we will chronicle in this book are not unique to our lineage, but have independently sprouted along numerous branches of earth’s evolutionary tree.

Since the days of Aristotle, scientists and philosophers have constructed what modern biologists refer to as a “scale of nature” (or, since scientists like using Latin terms, scala naturae). Aristotle created a hierarchy of all life-forms with humans being superior to other mammals, who were in turn superior to reptiles and fish, who were in turn superior to insects, who were in turn superior to plants.

Even after the discovery of evolution, the idea of a scale of nature continues to persist. This idea that there is a hierarchy of species is dead wrong. All species alive today are, well, alive; their ancestors survived the last 3.5 billion years of evolution. And thus, in that sense—the only sense that evolution cares about—all life-forms alive today are tied for first place.

Species fall into different survival niches, each of which optimizes for different things. Many niches—in fact, most niches—are better served by smaller and simpler brains (or no brains at all). Big-brained apes are the result of a different survival strategy than that of worms, bacteria, or butterflies. But none are “better.” In the eyes of evolution, the hierarchy has only two rungs: on one, there are those that survived, and on the other, those that did not.

Perhaps instead, one wants to define better by some specific feature of intelligence. But here still, the ranking will entirely depend on what specific intellectual skill we are measuring. An octopus has an independent brain in each of its tentacles and can blow a human away at multitasking. Pigeons, chipmunks, tuna, and even iguanas can process visual information

faster than a human. Fish have incredibly accurate real-time processing; have you ever seen how fast a fish whips through a maze of rocks if you try to grab it? A human would surely crash if he or she tried to move so quickly through an obstacle course.

My appeal: As we trace our story, we must avoid thinking that the complexification from past to future suggests that modern humans are strictly superior to modern animals. We must avoid the accidental construction of a scala naturae. All animals alive today have been undergoing evolution for the same amount of time.

However, there are, of course, things that make us humans unique, and because we are human, it makes sense that we hold a special interest in understanding ourselves, and it makes sense that we strive to make artificial humanlike intelligences. So I hope we can engage in a human-centered story without devolving into human chauvinism. There is an equally valid story to be told for any other animal, from honeybees to parrots to octopuses, with which we share our planet. But we will not tell these stories here. This book tells the story of only one of these intelligences: it tells the story of us.

an unremarkable hydrothermal vent. As boiling water burst from the seafloor, it smashed naturally occurring nucleotides together and transformed them into long molecular chains that closely resembled today’s DNA. These early DNA-like molecules were short-lived; the same volcanic kinetic energy that constructed them also inevitably ripped them apart. Such is the consequence of the second law of thermodynamics. That unbreakable law of physics which declares that entropy—the amount of disorder in a system—always and unavoidably increases; the universe cannot help but tend toward decay. After countless random nucleotide chains were constructed and destroyed, a lucky sequence was stumbled upon, one that marked, at least on Earth, the first true rebellion against the seemingly inexorable onslaught of entropy. This new DNA-like molecule wasn’t alive per se, but it performed the most fundamental process by which life would later emerge:

it duplicated itself.

Although these self-replicating DNA-like molecules also succumbed to the destructive effects of entropy, they didn’t have to survive individually to survive collectively—as long as they endured long enough to create their own copies, they would, in essence, persist. This is the genius of self-replication. With these first self-replicating molecules, a primitive version of the process of evolution began; any new lucky circumstances that facilitated more successful duplication would, of course, lead to more duplicates.

There were two subsequent evolutionary transformations that led to life. The first was when protective lipid bubbles entrapped these DNA molecules using the same mechanism by which soap, also made of lipids, naturally bubbles when you wash your hands. These DNA-filled microscopic lipid bubbles were the first versions of cells, the fundamental unit of life.

The second evolutionary transformation occurred when a suite of nucleotide-based molecules—ribosomes—began translating specific sequences of DNA into specific sequences of amino acids that were then folded into specific three-dimensional structures we call proteins. Once produced, these proteins float around inside a cell or are embedded in the wall of the cell fulfilling different functions. You have probably, at least in passing, heard that your DNA is made up of genes. Well, a gene is simply the section of DNA that codes for the construction of a specific and singular protein. This was the invention of protein synthesis, and it is here that the first sparks of intelligence made their appearance.

DNA is relatively inert, effective for self-duplication but otherwise limited in its ability to manipulate the microscopic world around it. Proteins, however, are far more flexible and powerful. In many ways, proteins are more machine than molecule. Proteins can be constructed and folded into many shapes—sporting tunnels, latches, and other robotic moving parts—and can thereby subserve endless cellular functions, including “intelligence.”

Even the simplest single-celled organisms—such as bacteria—have proteins designed for movement, motorized engines that convert cellular energy into propulsion, rotating propellers using a mechanism no less complex than the motor

of a modern boat. Bacteria also have proteins designed for perception—receptors that reshape when they detect certain features of the external environment, such as temperature, light, or touch.

Armed with proteins for movement and perception, early life could monitor and respond to the outside world. Bacteria can swim away from environments that lower the probability of successful replication, environments that have, for example, temperatures that are too hot or cold or chemicals that are destructive to DNA or cell membranes. Bacteria can also swim toward environments that are amenable to reproduction.

And in this way, these ancient cells indeed had a primitive version of intelligence, implemented not in neurons but in a complex network of chemical cascades and proteins.

The development of protein synthesis not only begot the seeds of intelligence but also transformed DNA from mere matter to a medium for storing information. Instead of being the self-replicating stuff of life itself, DNA was transformed into the informational foundation from which the stuff of life is constructed. DNA had officially become life’s blueprint, ribosomes its factory, and proteins its product.

With these foundations in place, the process of evolution was initiated in full force: variations in DNA led to variations in proteins, which led to the evolutionary exploration of new cellular machinery, which, through natural selection, were pruned and selected for based on whether they further supported survival. By this point in life’s story, we have concluded the long, yet-to-be-replicated, and mysterious process scientists call abiogenesis: the process by which nonbiological matter (abio) is converted into life (genesis).

The Terraforming of Earth

Shortly thereafter, these cells evolved into what scientists call the “last universal common ancestor,” or LUCA. LUCA was the genderless grandparent of all life; every fungus, plant, bacteria, and animal alive today, including us, descend from LUCA. It is no surprise, then, that all life shares the core features of LUCA: DNA, protein

synthesis, lipids, and carbohydrates.

LUCA, living around 3.5 billion years ago, likely resembled a simpler version of a modern bacteria. And indeed, for a long time after this, all life was bacterial. After a further billion years—through trillions upon trillions of evolutionary iterations—Earth’s oceans were brimming with many diverse species of these microbes, each with its own portfolio of DNA and proteins. One way in which these early microbes differed from one another was in their systems of energy production. The story of life, at its core, is as much about energy as it is about entropy.

Keeping a cell alive is expensive. DNA requires constant repair; proteins require perpetual replenishment; and cellular duplication requires a reconstruction of many inner structures. Hydrogen, an element abundant near hydrothermal vents, was likely the first fuel used to

finance these many processes. But this hydrogen-based energy system was inefficient and left life desperately grasping for enough energy to survive. After more than a billion years of life, this energetic poverty came to an end when a single species of bacteria—the cyanobacteria, also called

blue-green algae—found a far more profitable mechanism for extracting and storing energy: photosynthesis.

The most impressive biological system in these early cyanobacteria was not their protein factories or their products but their photosynthetic power plants—the structures that converted sunlight and carbon dioxide into sugar, which could then be stored and

converted into cellular energy. Photosynthesis was more efficient than prior cellular systems for extracting and storing energy. It provided cyanobacteria with an abundance of fuel with which to finance their duplication. Vast regions of the ocean quickly became covered in greenish gooey microbial mats—colonies of billions of cyanobacteria basking in the sun, sucking up carbon dioxide,

and endlessly reproducing.

Like most processes of energy production, from burning fossil fuels to harnessing nuclear fuels, photosynthesis

produced a pollutive exhaust. Instead of carbon dioxide or nuclear waste, the exhaust from photosynthesis was oxygen. Before this time, the Earth had no ozone layer. It was the cyanobacteria, with their newfound photosynthesis, that constructed Earth’s oxygen-rich atmosphere and began to terraform the planet from a gray volcanic rock to the oasis we know today. This event happened around 2.4 billion years ago, and it occurred so rapidly, at least in geological terms, that it has been called the Great Oxygenation Event. Over the course of one hundred million years,

oxygen levels skyrocketed. Unfortunately, this event was not a boon for all life. Scientists have a less forgiving name for it:

The Oxygen Holocaust.

Oxygen is an incredibly reactive element, which makes it dangerous in the carefully orchestrated chemical reactions of a cell. Unless special intracellular protective measures are taken, oxygen compounds will interfere with cellular processes, including the maintenance of DNA. This is why antioxidants—compounds that remove highly reactive oxygen molecules from your bloodstream—are believed to offer protection from cancer. The photosynthetic life-forms became victims of their own success, slowly suffocating in a cloud of their own waste. The rise of oxygen was followed by one of the deadliest

extinction events in Earth’s history.

As with many substances that are dangerous (uranium, gasoline, coal), oxygen can also be useful. This newly available element presented an energetic opportunity, and it was only a matter of time before life stumbled onto a way to exploit it. A new form of bacteria emerged that produced energy not from photosynthesis but from cellular respiration—the process by which oxygen and sugar is converted into energy, spewing out

carbon dioxide as exhaust. Respiring microbes began gobbling up the ocean’s excess oxygen and replenishing its depleted supply of carbon dioxide. What began as a pollutant to one form of life became fuel for another.

Life on Earth fell into perhaps the greatest symbiosis ever found between two competing but complementary systems of life, one that lasts to this day. One category of life was photosynthetic, converting water and carbon dioxide into sugar and oxygen. The other was respiratory, converting sugar and oxygen back into carbon dioxide. At the time, these two forms of life were similar, both single-celled bacteria. Today this symbiosis is made up of very different forms of life. Trees, grass, and other plants are some of our modern photosynthesizers, while fungi and animals are some of our modern respirators.

Cellular respiration requires sugar to produce energy, and this basic need provided the energetic foundation for the eventual intelligence explosion that occurred uniquely within the descendants of respiratory life. While most, if not all, microbes at the time exhibited primitive levels of intelligence, it was only in respiratory life that intelligence was later elaborated and extended. Respiratory microbes differed in one crucial way from their photosynthetic cousins: they needed to hunt. And hunting required a whole new degree of smarts.

Three Levels

The ecosystem two billion years ago was not a particularly war-torn world.

However, unlike the cells that came before, respiratory life could survive only by stealing the energetic prize—the sugary innards—of photosynthetic life. Thus, the world’s utopic peace ended quite abruptly with the arrival of aerobic respiration. It was here that microbes began to actively eat other microbes. This fueled the engine of evolutionary progress; for every defensive innovation prey evolved to stave off being killed, predators evolved an offensive innovation to overcome that same defense. Life became caught in an arms race, a perpetual feedback loop: offensive innovations led to defensive innovations that required further offensive innovations.

Out of this maelstrom, a large diversification of life emerged. Some species remained small single-celled microbes. Other species evolved into the first eukaryotes (pronounced “you-care-ee-oats”), cells that were over a hundred times larger, produced a thousand times more energy, and had

much more internal complexity. These eukaryotes were the most advanced

microbial killing machines yet. Eukaryotes were the first to evolve phagotrophy—the hunting strategy of literally engulfing other cells and breaking them down inside their cellular walls. These eukaryotes, armed with more energy and complexity, further diversified into the first plants, the first fungi, and the precursors of the first animals. The fungi and animal descendants of eukaryotes retained their need to hunt (they were respirators), while plant lineage returned to a lifestyle of photosynthesis.

What was common across these eukaryote lineages was that all three—plants, fungi, and animals—each independently evolved multicellularity. Most of what you see and think of as life—humans, trees, mushrooms—are primarily multicellular organisms, cacophonies of billions of individual cells all working together to create a singular emergent organism. A human is made up of exactly such diverse types of specialized cells: skin cells, muscle cells, liver cells, bone cells, immune cells, blood cells. A plant has specialized cells too. These cells all serve different functions while still serving a common purpose: supporting the survival of the overall organism.

And so, seaweed-like underwater plants began sprouting,

mushroom-like fungi began growing, and primitive animals began slowly swimming around. By about eight hundred million years ago, life would have fallen into three broad levels of complexity. At level one, there was single-celled life, made up of microscopic bacteria and single-celled eukaryotes. At level two, there was small multicellular life, large enough to engulf single-celled organisms but small enough to move around using basic cellular propellers. At level three, there was large multicellular life; too big to move with cellular propellers, and therefore forming immobile structures.

These early animals probably wouldn’t resemble what you think of as animals. But they contained something that made them different from all other life at the time: neurons.

The Neuron

What a neuron is and what it does depends on whom you ask. If you ask a biologist, neurons are the primary cells that make up the nervous system. If you ask a machine learning researcher, neurons are the fundamental units of neural networks, little accumulators that perform the basic task of computing a weighted summation of their inputs. If you ask a psychophysicist, neurons are the sensors that measure features of the external world. If you ask a neuroscientist specializing in motor control, neurons are effectors, the controllers of muscles and movement. If you ask other people, you might get a wide range of answers, from “Neurons are little electrical wires in your head” to “Neurons are the stuff of consciousness.” All of these answers are right, carrying a kernel of the whole truth, but incomplete on their own.

The nervous systems of all animals—from worms to wombats—are made up of these stringy odd-shaped cells called neurons. There is an incredible diversity of neurons, but despite this diversity in shapes and sizes, all neurons work the same way. This is the most shocking observation when comparing neurons across species—they are all, for the most part, fundamentally identical. The neurons in the human brain operate the same way as the neurons in a jellyfish. What separates you from an earthworm is not the unit of intelligence itself—neurons—but how these units are wired together.

Animals with neurons share a common ancestor, an organism in whom the first neurons evolved and from

whom all neurons descend. It seems that in this ancient grandmother of animals, neurons attained their modern form; from this point on, evolution rewired neurons but made no meaningful adjustments to the basic unit itself. This is a glaring example of how prior innovations impose constraints on future innovations, often leaving early structures unchanged—the fundamental building blocks of brains have been essentially the same for over six hundred million years.

Why Fungi Don’t Have Neurons but Animals Do

You aren’t all that different from mold. Despite their appearance, fungi have more in common with animals than they do with plants. While plants survive by photosynthesis, animals and fungi both survive by respiration. Animals and fungi both breathe oxygen and eat sugar; both digest their food, breaking cells down using enzymes and absorbing their inner nutrients; and both share a much more recent common ancestor than either do with plants, which diverged much earlier. At the dawn of multicellularity, fungi and animal life would have been extremely similar. And yet one lineage (animals) went on to evolve neurons and brains, and the other (fungi) did not. Why?

Sugar is produced only by life, and thus there are only two ways for large multicellular respiratory organisms to feed. One is to wait for life to die, and the other is to catch and kill living life. Early in the fungi-animal divergence, they each settled into opposing feeding strategies. Fungi chose the strategy of waiting, and animals chose the strategy of killing.

Fungi produce trillions of single-celled spores that float around dormant. If by luck one happens to find itself near dying life, it will blossom into a large fungal structure, growing hairy filaments into the decaying tissue, secreting enzymes, and absorbing the released nutrients. This is why mold always shows up in old food. Fungal spores are all around us, patiently waiting for something to die. Fungi are currently, and likely have always been, Earth’s garbage collectors.

Early animals, however, settled into a strategy of actively catching and ingesting level-two multicellular prey (see

as much as jellyfish embryos. While every animal develops this way, no other kingdom of life does this. This provides a glaring hint as to the evolutionary template on which all animals derive: we form stomachs for ingesting food. All animals that engage in such gastrulation also have neurons and muscles and seem to derive from a common

neuron-enabled animal ancestor. Gastrulation, neurons, and muscles are the three inseparable features that bind all animals together and separate animals from all other kingdoms of life.

Some have gone so far as to argue that the grandmother of animals was literally a little

gastrula-shaped creature with neurons. But this is a scientific arena full of controversy—just because all animals develop this way doesn’t mean that they ever truly lived in this form.

Another interpretation, supported by fossils, suggests that the first animals may have been

similar to today’s corals. To the naked eye, a coral is so simple that it doesn’t look all that different from a fungus or a plant (

Figure 1.7: Tree of neuron-enabled animals

Original art by Rebecca Gelernter

Figure 1.8: Soft coral as a model for early animal life

Original art by Rebecca Gelernter

false starts and wrong turns. As far back as the famous ancient Greek physician Hippocrates in 400 BCE, it has been known that animals contain a system of stringy material later called nerves (from the Latin word nervus, meaning “sinew”), flowing to and from the brain, and that this system is the medium by which muscles are controlled and sensations are perceived. They knew this through horrifying experiments of severing spinal cords and clamping nerves in living pigs and other livestock. But the ancient Greeks wrongly concluded that it was “animal spirits” that flowed through these nerves. It would be many centuries before this error was corrected. More than two thousand years later, even the great Isaac Newton would speculate, incorrectly, that nerves communicated via vibrations flowing through a nerve fluid he called ether. It wasn’t until the late 1700s that scientists discovered that flowing through the nervous system was

not ether but electricity. Electrical current applied to a nerve will flow toward downstream muscles and cause them to contract.

But many errors remained. At the time, it was thought that the nervous system was made up of a single web of nerves, analogous to the circulatory system’s homogenous web of blood vessels. It wasn’t until the end of the nineteenth century, with more advanced microscopes and staining techniques, that scientists discovered that the nervous system was made up of independent cells—neurons—that, although wired together, are separate and

generate their own signals. This also revealed that electrical signals flow only in a single direction within a neuron, from one part that takes inputs, called dendrites, to another that sends electrical output, called the axon. This output flows to other neurons or other types of cells (such as muscle cells) to activate them.

In the early 1920s, a young English neurologist by the name of Edgar Adrian returned to Cambridge University after a long stint of medical service during World War I. Adrian, like many researchers at the time, was interested in electrically recording neurons to decipher how and what they communicated. The problem had always been that electrical recording devices were too large and crude to record the activity of a single neuron and hence always produced a noisy mess of signals from multiple neurons. Adrian and his collaborators were the first to find a technical solution to this problem, inventing the field of single-neuron electrophysiology. This gave scientists, for the first time, a view into the language of individual neurons. The subsequent three discoveries earned

Adrian the Nobel Prize.

spikes or action potentials. A neuron is either on or off; there is no in between. In other words, neurons act less like an electric power line with a constant flow of electricity and more like an electric telegraph cable, with patterns of electrical clicks and pauses. Adrian himself noted the similarity between neural spikes and Morse code.

This discovery of spikes presented a puzzle for Adrian. You can clearly perceive stimulus strengths in your senses—you can discriminate between different volumes of sound, brightness of light, potency of smells, severity of pain. How could a simple binary signal that was either on or off communicate a numerical value, such as the graded strengths of stimuli? The realization that the language of neurons was spikes didn’t tell researchers much about what a sequence of spikes meant. Morse code is, well, a code—it is an efficient trick to store and transmit information across a single electrical wire. Adrian was the first scientist to use the word information to refer to

the signals of neurons, and he devised a simple experiment to try to decode them.

Adrian took a muscle from the neck of a deceased frog and attached a recording device to a single stretch-sensing neuron in the muscle. Such neurons have receptors that are stimulated when muscles are stretched. Adrian then attached various weights to the muscle. The question was: How would the responses of these stretch-sensing neurons change based on the weight placed on the muscle?

of the spike itself. Since Adrian’s initial work, such rate coding has been found in neurons across the animal kingdom,

from jellyfish to humans. Touch-sensitive neurons encode pressure

in their firing rate; photosensitive neurons encode contrast in their firing rate; smell-sensitive neurons encode

concentration in their firing rate. The neural code for movement is also a firing rate: the faster the spikes of the neurons that stimulate muscles, the greater the contraction

force of the muscles. This is how you are able to delicately pet your dog and also pick up fifty-pound weights—if you couldn’t modulate the strength of muscle contractions, you wouldn’t be very pleasant to be around.

Adrian’s third discovery was the most surprising of all. There is a problem with trying to translate natural variables, such as the pressure of touch or the brightness of light into this language of rate codes. The problem is this: these natural variables have a massively larger range than can be encoded in the firing rate of a neuron.

Take vision, for example. What you don’t realize (because your sensory machinery abstracts it away) is that the luminance of light is varying astronomically all around you. The amount of light entering your eyes when you look at a white piece of paper is one million times greater if you are in sunlight than if you are in moonlight.

a page in moonlight!

This is not just a feature of light; all sensory modalities, from smell to touch to sound, require the discrimination of hugely varying natural variables. This wouldn’t necessarily be a problem except for a big limitation of neurons—for a variety of biochemical reasons, it is simply impossible for a neuron to fire faster than around

five hundred spikes per second. This means that a neuron needs to encode a range of natural variables that can vary by a factor of over a million all within a firing rate ranging only from 0 to 500 spikes per second. This could be reasonably called the “squishing problem”: neurons have to squish this huge range of natural variables into a comparably minuscule range of firing rates.

This makes rate coding, on its own, untenable. Neurons simply cannot directly encode such a wide range of natural variables in such a small range of firing rates without losing a huge amount of precision. The resulting imprecision would make it impossible to read inside, detect subtle smells, or notice a soft touch.

It turns out that neurons have a clever solution to this problem. Neurons do not have a fixed relationship between natural variables and firing rates. Instead, neurons are always adapting their firing rates to their environment; they are constantly remapping the relationship between variables in the natural world and the language of firing rates. The term neuroscientists use to describe this observation is adaptation; this was Adrian’s third discovery.

In Adrian’s frog muscle experiments, a neuron might fire one hundred spikes in response to a certain weight. But after this first exposure, the neuron quickly adapts; if you apply the same weight shortly after, it might elicit only eighty spikes. And as you keep doing this, the number of spikes continues to decline. This applies in many neurons throughout the brains of animals—the stronger the stimuli, the greater the change in the neural threshold for spiking. In some sense, neurons are more a measurement of relative changes in stimulus strengths, signaling how much the strength of a stimulus changed relative to its baseline as opposed to signaling the absolute value of the stimulus.

Here’s the beauty: Adaptation solves the squishing problem. Adaptation enables neurons to precisely encode a broad range of stimulus strengths despite a limited range of firing rates. The stronger a stimulus is, the more strength will be required to get the neuron to respond similarly next time. The weaker a stimulus is, the more sensitive neurons become.

The late nineteenth and early twentieth centuries were filled with rich discoveries about the inner workings of neurons. A long roster of giants in neuroscience emerged in this period, leading to a platter of Nobel Prizes, not only Edgar Adrian, but also Santiago Ramón y Cajal, Charles Sherrington, Henry Dale,

John Eccles, and others. One important discovery was that nerve impulses pass from one neuron to another by way of synapses, which are microscopic gaps between neurons. Spikes in the input neuron trigger the release of chemicals called neurotransmitters, which travel across the synapse over the course of nanoseconds, attach to a bunch of protein receptors, trigger ions to flow into the target neuron, and thereby change its charge. While neural communication within a neuron is electrical, across neurons, it is chemical.

Figure 1.12

Original art by Rebecca Gelernter

and another must relax. And the converse for closing its mouth. The existence of both excitatory and inhibitory neurons enabled the first neural circuits to implement a form of logic required for reflexes to work. They can enforce the rule of “do this, not that,” which was perhaps the first glimmer of intellect to emerge from circuits of neurons. “Do this, not that” logic was not new—such logic already existed in the protein cascades of single cells. But this ability was recapitulated in the medium of neurons, which made such logic newly possible on the scale of level-three multicellular life. Inhibitory neurons enabled the necessary inner logic for catch-and-

swallow reflexes to work.

While the first animals, whether gastrula-like or polyp-like creatures, clearly had neurons, they had no brain. Like today’s coral polyps and jellyfish, their nervous system was what scientists call a nerve net: a distributed web of independent neural circuits implementing their own independent reflexes.

But with the evolutionary feedback loop of predator-prey in full force, with the animal niche of active hunting, and the building blocks of neurons in place, it was only a matter of time before evolution stumbled on

Figure 2.2

Original art by Rebecca Gelernter

Figure 2.3: Why bilateral symmetry is better for navigation

Original art by Rebecca Gelernter

a grain of rice. Evidence suggests that they first emerged sometime in the Ediacaran period, an era that stretched from 635 to 539 million years ago. The seafloor of the Ediacaran was filled with thick green gooey microbial mats in its shallow areas—vast colonies of cyanobacteria basking in the sun. Sensile multicellular animals like corals, sea sponges, and early plants would have been common.

Modern nematodes are believed to have remained relatively unchanged since early bilaterians; these creatures give us a window into the inner workings of our wormlike ancestors. Nematodes are almost literally just the basic template of a bilaterian: not much more than a head, mouth, stomach, butt, some muscles, and a brain.

The first brains were, like those of nematodes,

almost definitely very simple. The most well-studied nematode, Caenorhabditis elegans, has only 302 neurons, a minuscule number compared to

a human’s 85 billion. However, nematodes display remarkably sophisticated behavior despite their minuscule brains. What a nematode does with their hopelessly simple brain suggests what the first bilaterians did with theirs.

The most obvious behavioral difference between nematodes and more ancient animals like corals is that nematodes spend a lot of time moving. Here’s an experiment: Put a nematode on one side of a petri dish, place a tiny piece of food on the other side. Three things would reveal themselves: First, it always

finds the food. Second, it finds the food much faster than it would if it were simply moving around randomly. And third, it doesn’t swim directly toward the food but rather circles

in on the food.

The worm is not using vision; nematodes can’t see. They have no eyes to render any image useful for navigation. Instead, the worm is using smell. The closer it gets to the source of a smell, the higher the concentration of that smell. Worms exploit this fact to find food. All a worm must do is turn toward the direction where the concentration of food particles is increasing, and away from the direction it is decreasing. It is quite elegant how simple yet effective this navigational strategy is. It can be summarized in two rules:

1. If food smells increase, keep going forward.
2. If food smells decrease, turn.

This was the breakthrough of steering. It turns out that to successfully navigate in the complicated world of the ocean floor, you don’t actually need an understanding of that two-dimensional world. You don’t need an understanding of where you are, where food is, what paths you might have to take, how long it might take, or really anything meaningful about the world. All you need is a brain that steers a bilateral body toward increasing food smells and away from decreasing food smells.

Steering can be used not only to navigate toward things but to navigate away from things. Nematodes have sensory cells that detect light, temperature, and touch. They steer away from light, where predators can more easily see them; they steer away from noxious heat and cold, where their bodily functions become harder to perform; and they steer away from surfaces that are sharp, where their fragile

bodies might get wounded.

This trick of navigating by steering was not new. Single-celled organisms like bacteria navigate around their environments in a similar way. When a protein receptor on the surface of a bacterium detects a stimulus like light, it can trigger a chemical process within the cell that changes the movement of the cell’s protein propellers, thereby causing it to change its direction. This is how single-celled organisms like bacteria swim toward food sources or away from dangerous chemicals. But this mechanism works only on the scale of individual cells, where simple protein propellers can successfully reorient the entire life-form. Steering in an organism that contains millions of cells required a whole new setup, one in which a stimulus activates circuits of neurons and the neurons activate muscle cells, causing specific turning movements. And so the breakthrough that came with the first brain was not steering per se, but steering on the scale of multicellular organisms.

The First Robot

In the 1980s and 1990s a schism emerged in the artificial intelligence community. On one side were those in the symbolic AI camp, who were focused on decomposing human intelligence into its constituent parts in an attempt to imbue AI systems with our most cherished skills: reasoning, language, problem solving, and logic. In opposition were those in the behavioral AI camp, led by the roboticist Rodney Brooks at MIT, who believed the symbolic approach was doomed to fail because “we will never understand how to decompose human level intelligence until we’ve had a lot of practice with simpler level intelligences.”

Brooks’s argument was partly based on evolution: it took billions of years before life could simply sense and respond to its environment; it took another five hundred million years of tinkering for brains to get good at motor skills and navigation; and only after all of this hard work did language and logic appear. To Brooks, compared to how long it took for sensing and moving to evolve, logic and language appeared in a blink of an eye. Thus he concluded that “language . . . and reason, are all pretty simple once the essence of being and reacting are available. That essence is the ability to move around in a dynamic environment, sensing the surroundings to a degree sufficient to achieve the necessary maintenance of life and reproduction. This part of intelligence is where evolution has concentrated its time—it is much harder.”

To Brooks, while humans “provided us with an existence proof of [human-level intelligence], we must be careful about what the lessons are to

be gained from it.” To illustrate this, he offered a metaphor:

Suppose it is the 1890s. Artificial flight is the glamor subject in science, engineering, and venture capital circles. A bunch of [artificial flight] researchers are miraculously transported by a time machine to the 1990s for a few hours. They spend the whole time in the passenger cabin of a commercial passenger Boeing 747 on a medium duration flight. Returned to the 1890s they feel invigorated, knowing that [artificial flight] is possible on a grand scale. They immediately set to work duplicating what they have seen. They make great progress in designing pitched seats, double pane windows, and know that if only they can figure out those weird “plastics” they will have the

grail within their grasp.

By trying to skip simple planes and directly build a 747, they risked completely misunderstanding the principles of how planes work (pitched seats, paned windows, and plastics are the wrong things to focus on). Brooks believed the exercise of trying to reverse-engineer the human brain suffered from this same problem. A better approach was to “incrementally build up the capabilities of intelligence systems, having complete systems at each step.” In other words, to start as evolution did, with simple brains, and add complexity from there.

Many do not agree with Brooks’s approach, but whether you agree with him or not, it was Rodney Brooks who, by any reasonable account, was the first to build a commercially successful domestic robot; it was Brooks who made the first small step toward Rosey. And this first step in the evolution of commercial robots has parallels to the first step in the evolution of brains. Brooks, too, started with steering.

In 1990, Brooks cofounded a robotics company named iRobot, and in 2002, he introduced the Roomba, a vacuum-cleaner robot. The Roomba was a robot that autonomously navigated around your house vacuuming the floor. It was an immediate hit; new models are still being produced today, and iRobot has sold

over forty million units.

The first Roomba and the first bilaterians share a surprising number of properties. They both had extremely simple sensors—the first Roomba could detect only a handful of things, such as when it hit a wall and when it was close to its charging base. They both had simple brains—neither used the paltry sensory input they received to build a map of their environment

or to recognize objects. They both were bilaterally symmetric—the Roomba’s wheels allowed it to go forward and backward only. To change directions, it had to turn in place and then resume its forward movement.

The Roomba could clean all the nooks and crannies of your floor by simply moving around randomly, steering away from obstacles when it bumped into them, and steering toward its charging station when it was low on battery. Whenever the Roomba hit a wall, it would perform a random turn and try to move forward again. When it was low on battery, the Roomba searched for a signal from its charging station, and when it detected the signal, it simply turned in the direction where the signal was strongest, eventually making it back to its charging station.

The navigational strategies of the Roomba and first bilaterians were not identical. But it may not be a coincidence that the first successful domestic robot contained an intelligence not so unlike the intelligence of the first brains. Both used tricks that enabled them to navigate a complex world without actually understanding or modeling that world.

While others remained stuck in the lab working on million-dollar robots with eyes and touch and brains that attempted to compute complicated things like maps and movements, Brooks built the simplest possible robot, one that contained hardly any sensors and that computed barely anything at all. But the market, like evolution, rewards three things above all: things that are cheap, things that work, and things that are simple enough to be discovered in the first place.

While steering might not inspire the same awe as other intellectual feats, it was surely energetically cheap, it worked, and it was simple enough for evolutionary tinkering to stumble upon it. And so it was here where brains began.

Valence and the Inside of a Nematode’s Brain

Around the head of a nematode are sensory neurons, some of which respond to light, others to touch, and others to specific chemicals. For steering to work, early bilaterians needed to take each smell, touch, or other stimulus they detected and make a choice: Do I approach this thing, avoid this thing, or ignore this thing?

The breakthrough of steering required bilaterians to categorize the world into things to approach (“good things”) and things to avoid (“bad things”). Even a Roomba does this—obstacles are bad; charging station when low on battery is good. Earlier radially symmetric animals did not navigate, so they never had to categorize things in the world like this.

When animals categorize stimuli into good and bad, psychologists and neuroscientists say they are imbuing stimuli with valence. Valence is the goodness or badness of a stimulus. Valence isn’t about a moral judgment; it’s something far more primitive: whether an animal will respond to a stimulus by approaching it or avoiding it. The valence of a stimulus is, of course, not objective; a chemical, image, or temperature, on its own, has no goodness or badness. Instead, the valence of a stimulus is subjective, defined only by the brain’s evaluation of its goodness or badness.

How does a nematode decide the valence of something it perceives? It doesn’t first observe something, ponder it, then decide its valence. Instead, the sensory neurons around its head directly signal the stimulus’s valence. One group of sensory neurons are, effectively, positive-valence neurons; they are directly activated by things nematodes deem good (such as food smells). Another group of sensory neurons are, effectively, negative-valence neurons; they are directly activated by things nematodes deem bad (such as high temperatures, predator smells, bright light).

In nematodes, sensory neurons don’t signal objective features of the surrounding world—they encode steering votes for how much a nematode wants to steer toward or away from something. In more complex bilaterians, such as humans, not all sensory machinery is like this—the neurons in your eyes detect features of images; the valence of the image is computed elsewhere. But it seems that the first brains began with sensory neurons that didn’t care to measure objective features of the world and instead cast the entirety of perception through the simple binary lens of valence.

to different downstream neurons.

Consider how a nematode uses this circuit to find food. Nematodes have positive-valence neurons that trigger forward movement when the concentration of a food smell increases. As we saw in the sensory neurons in the nerve net of earlier animals, these neurons quickly adapt to baseline levels of smells. This enables these valence neurons to signal changes across a wide range of smell concentrations. These neurons will generate a similar number of spikes whether a smell concentration goes from two to four parts or from one hundred to two hundred parts. This enables valence neurons to keep nudging the nematode in the right direction. It is the signal for Yes, keep going! from the first whiff of a faraway food smell all the way to the food source.

This use of adaptation is an example of evolutionary innovations enabling future innovations. Steering toward food in early bilaterians was possible only because adaptation had already evolved in earlier radially symmetric animals. Without adaptation, valence neurons would be either too sensitive (and continuously misfire when smells are too close) or not sensitive enough (unable to detect faraway smells).

At this point, new navigational behaviors could emerge simply by modifying the conditions under which different valence neurons get excited. For example, consider how nematodes navigate toward optimal temperatures. Temperature navigation requires some additional cleverness relative to the simple steering toward smells: the decreasing concentration of a food smell is always bad, but the decreasing temperature of an environment is bad only if a nematode is already too cold. If a nematode is hot, then decreasing temperature is good. A warm bath is miserable in a scorching summer but heavenly in a cold winter. How did the first brains manage to treat temperature fluctuations differently depending on the context?

Nematodes have a negative-valence neuron that triggers turning when temperatures increase, but only if the temperature is already above a certain threshold; it is a Too hot! neuron. Nematodes also have a Too cold! neuron; it triggers turning when temperatures decrease, but only when temperatures are already below a certain threshold. Together, these two negative-valence neurons enable nematodes to quickly steer away from heat when they’re too hot and away from

cold when they’re too cold. Deep in the human brain is an ancient structure called the hypothalamus that houses temperature-sensitive neurons that work in the same way.

The Problem of Trade-Offs

Steering in the presence of multiple stimuli presented a problem: What happens if different sensory cells vote for steering in opposite directions? What if a nematode smells both something yummy and something dangerous at the same time?

Scientists have tested nematodes in exactly such situations. Put a bunch of nematodes on one side of a petri dish and place yummy food on the opposite side of the petri dish, then place a dangerous copper barrier (nematodes hate copper) in the middle. Nematodes then have a problem: Are they willing

to cross the barrier to get to the food? Impressively, the answer is—as you would expect from an animal with even an iota of smarts—it depends. It depends on the relative concentration of the food smell versus the copper smell.

At low levels of copper, most nematodes cross the barrier; at intermediate levels of copper, only some do; at high levels of copper, no nematodes are willing to cross the barrier.

This ability to make trade-offs in the decision making process has been tested across different species of simple wormlike bilaterians and

across different sensory modalities. The results consistently show that even the simplest brains, those with less than a thousand neurons, can make these trade-offs.

This requirement of integrating input across sensory modalities was likely one reason why steering required a brain and could not have been implemented in a distributed web of reflexes like those in a coral polyp. All these sensory inputs voting for steering in different directions had to be integrated together in a single place to make a single decision; you can go in only one direction at a time. The first brain was this mega-integration center—one big neural circuit in which steering directions were selected.

You can get an intuition for how this works from

This is another example of how past innovations enabled future innovations. Just as a bilaterian cannot both go forward and turn at the same time, a coral polyp cannot both open and close its mouth at the same time. Inhibitory neurons evolved in earlier coral-like animals to enable these mutually exclusive reflexes to compete with each other so that only one reflex could be selected at a time; this same mechanism was repurposed in early bilaterians to enable them to make trade-offs in steering decisions. Instead of deciding whether to open or close one’s mouth, bilaterians used inhibitory neurons to decide whether to go forward or turn.

Are You Hungry?

The valence of something depends on an animal’s internal state. A nematode’s choice of whether to cross a copper barrier to get to food depends not only on the relative level of food smell and copper but also on how

hungry a nematode is. It won’t cross the barrier to get to food if it is full, but it will if it is hungry. Further, nematodes can completely flip their preferences depending on

how hungry they are. If a nematode is well fed, it will steer away from carbon dioxide; if hungry, it will steer toward it. Why? Carbon dioxide is a chemical that is released by both food and predators, so when a nematode is full, pursuing carbon dioxide for food isn’t worth the risk of predators; when it is hungry, however, the chance that the carbon dioxide is signaling food, not predators, makes it worth the risk.

The brain’s ability to rapidly flip the valence of a stimulus depending on internal states is ubiquitous. Compare the salivary ecstasy of the first bite of your favorite dinner after a long day of skipped meals to the bloated nausea of the last bite after eating yourself into a food coma. Within mere minutes, your favorite meal can transform from God’s gift to mankind to something you want nowhere near you.

The mechanisms by which this occurs are relatively simple and shared across bilaterians. Animal cells release specific chemicals—“full signals” such as insulin—in response to having a

healthy amount of energy. And animal cells release a different set of chemicals—hunger signals—in response to having insufficient amounts of energy. Both signals diffuse throughout an animal’s body and provide a persistent global signal for an animal’s level of hunger. The sensory neurons of nematodes have receptors that detect the presence of these signals and change their responses accordingly. Positive valence food smell neurons in C. elegans become more responsive to food smells in the presence of hunger signals and less responsive in the presence of full signals.

Internal states are present in a Roomba as well. A Roomba will ignore the signal from its home base when it is fully charged. In this case, the signal from the home base can be said to have neutral valence. When the Roomba’s internal state changes to one where it is low on battery, the signal from home base shifts to having positive valence: the Roomba will no longer ignore the signal from its charging station and will steer toward it to replenish its battery.

Steering requires at least four things: a bilateral body plan for turning, valence neurons for detecting and categorizing stimuli into good and bad, a brain for integrating input into a single steering decision, and the ability to modulate valence based on internal states. But still, evolution continued tinkering. There is another trick that emerged in early bilaterian brains, a trick that further bolstered the effectiveness of steering. That trick was the early kernel of what we now call emotion.

separate words for each. Which language best differentiates the objective categories of emotional states produced by brains? Many careers have been spent hunting for these objective categories in human brains; today, most neuroscientists believe that such objective categorizations do not exist, at least not at the level of words like sehnsucht or grief. Instead, it seems such emotion categories are largely culturally learned. We will find out more about how this works in later breakthroughs. For now, we begin by asking about the simpler origins of emotions. The basic template of emotions evolved as an intellectual trick to solve a specific set of problems faced by the first brains. And so we begin with the simplest two features of emotions, those that are universal not only across human cultures but also across the animal kingdom, those features of emotions that we inherited from the first brains: valence and arousal.

Neuroscientists and psychologists use the word affect to refer to these

two attributes of emotions; at any given point, humans are in an affective state represented by a location across these two dimensions of valence and arousal. While rigorous definitions of categories of human emotions themselves elude philosophers, psychologists, and neuroscientists alike, affect is the relatively well accepted unifying foundation of emotion.

The universality of affect can be seen in our intuitions; it is easy to take suites of nuanced emotion words—calm, elated, tense, upset, depressed, bored—and assign them to the affective states from which they derive (see

adrenaline, and blood pressure. And there are clear neurophysiological signatures that differentiate levels of valence, such as stress-hormone levels, dopamine levels, and the activation

of specific brain regions. And while cultures around the world differ in their classifications of specific emotion categories, such as anger and fear, classifications of affective states are quite universal. All cultures have words to communicate the concepts

of valence and arousal, and newborn children across cultures have universal facial and body signatures for arousal and valence

(e.g., crying and smiling).

The universality of affect stretches beyond the bounds of humanity; it is found across the animal kingdom. Affect is the ancient seed from which modern emotions sprouted. But why did affect evolve?

Steering in the Dark

Even nematodes with their minuscule nervous systems have affective states, albeit incredibly simple ones. Nematodes express different levels of arousal: When well fed, stressed, or ill, they hardly move at all and become unresponsive to external stimuli (low arousal); when hungry, detecting food, or sniffing predators, they will continually swim around (high arousal). The affective states of nematodes also express different levels of valence. Positive-valenced stimuli facilitate feeding, digestion, and reproductive activities (a primitive good mood), while negative-valenced stimuli inhibit all of these (a primitive bad mood).

Put these different levels of arousal and valence together and you get a primitive template of affect. Negative-valenced stimuli trigger a behavioral repertoire of fast swimming and infrequent turns, which can be thought of as the most primitive version of an aroused bad mood (which is often called the state of escaping), while the detection of food triggers a repertoire of slow swimming and frequent turns, which can be thought of as the most primitive version of an aroused good mood (which is often called the state of exploiting). Escaping leads worms to rapidly change location; exploiting leads worms to search their local surroundings (to exploit their surroundings for food). Although nematodes don’t share the same complexity of emotions as humans—they do not know the rush of young love or the bittersweet tears of sending a child off to college—they clearly show the basic template of affect. These incredibly simple affective states of nematodes offer a clue as to why affect evolved in the first place.

Suppose you put an unfed nematode in a big petri dish with a hidden patch of food. Even if you obscure any food smell for the nematode to steer toward, the nematode won’t just dumbly sit waiting for a whiff of food. The nematode will rapidly swim and relocate itself; in other words, it will escape. It does this because one thing that triggers escape is hunger. When the nematode happens to stumble upon the hidden food, it will immediately slow down and start rapidly turning, remaining in the same general location that it found food—it will shift from escaping to exploiting. Eventually, after eating enough food, the worm will stop moving and become immobile and unresponsive. It will shift to satiation.

Scientists tend to shy away from the term affective states in simple bilaterians such as nematodes and instead use the safer term behavioral states; this avoids the suggestion that nematodes are, in fact, feeling anything. Conscious experience is a philosophical quagmire we will only briefly touch on later. Here, at least, this issue can be sidestepped entirely; the conscious experience of affect—whatever it is and however it works—likely evolved after the raw underlying mechanisms of affect. This can be seen even in humans—the parts of the human brain that generate the experience of negative or positive affective states are evolutionarily newer and distinct from the parts of the brain that generate the reflexive avoidance and approach responses.

The defining feature of these affective states is that, although often triggered by external stimuli, they persist for long after the stimuli are gone. This feature of affective states stretches all the way from nematodes to humans—just as a nematode remains in a fear-like state for many minutes after a single sniff of a predator, a human mood can be soured for hours after a single unfriendly social interaction. The benefit of this persistence, at least at first, is unclear. If anything, it seems a bit dumb—nematodes keep trying to escape even after predators are long gone and they keep trying to exploit a local area for food even after all

the food is gone. The idea that the function of the first brain was for steering provides a clue as to why nematodes have—and why the first bilaterians likely also had—these states: such persistence is required for steering to work.

Sensory stimuli, especially the simple ones detected by nematodes, offer transient clues, not consistent certainties, of what exists in the real world. In the wild, outside of a scientist’s petri dish, food does not make perfectly distributed smell gradients—water currents can distort or even completely obscure smells, disrupting a worm’s ability to steer toward food or away from predators. These persistent affective states are a trick to overcome this challenge: If I detect a passing sniff of food that quickly fades, it is likely that there is food nearby even if I no longer smell it. Therefore, it is more effective to persistently search my surroundings after encountering food, as opposed to only responding to food smells in the moment that they are detected. Similarly, a worm passing through an area full of predators won’t experience a constant smell of predators but rather catch a transient hint of one nearby; if a worm wants to escape, it is a good idea to persistently swim away even after the smell has faded.

Like a pilot trying to fly a plane while looking through an opaque or obscured window, she would have no choice but to learn to fly in the darkness, using only the clues offered by the flickers of the outside world. Similarly, worms had to evolve a way to “steer in the dark”—to make steering decisions in the absence of sensory stimuli. The first evolutionary solution was affect, behavioral repertoires that can be triggered by external stimuli but persist long after they have faded.

This feature of steering shows up even in a Roomba. Indeed, Roombas were designed to have different behavioral states for the same reason. Normally, they explore rooms by moving around randomly. However, if a Roomba encounters a patch of dirt, it activates Dirt Detect, which changes its repertoire; it begins turning in circles in a local area. This new repertoire is triggered by the detection of dirt but persists for a time even after dirt is no longer detected. Why was the Roomba designed to do this? Because it works—detecting a patch of dirt in one location is predictive of nearby patches of dirt. Thus, a simple rule to improve the speed of getting all the dirt is to shift toward local search for a time after detecting dirt. This is exactly the same reason nematodes evolved to shift their behavioral state from exploration to exploitation after encountering food and locally search their surroundings.

Dopamine and Serotonin

The brain of a nematode generates these affective states using chemicals called neuromodulators. Two of the most famous neuromodulators are dopamine and serotonin. Antidepressants, antipsychotics, stimulants, and psychedelics all exert their effects by manipulating these neuromodulators. Many psychiatric conditions, including depression, obsessive-compulsive disorder, anxiety, post-traumatic stress disorder, and schizophrenia are believed to be caused, at least in part, by imbalances in neuromodulators. Neuromodulators evolved long before humans appeared; they began their connection to affect as far back as the first bilaterians.

Unlike excitatory and inhibitory neurons, which have specific, short-lived effects on only the specific neurons they connect to, neuromodulatory neurons have subtle, long-lasting, and wide-ranging effects on many neurons. Different neuromodulatory neurons release different neuromodulators—dopamine neurons release dopamine, serotonin neurons release serotonin. And neurons throughout an animal’s brain have different types of receptors for different types of neuromodulators—neuromodulators can gently inhibit some neurons while simultaneously activating others; they can make some neurons more likely to spike while making others less likely to spike; they can make some neurons more sensitive to activation while dulling the responses of others. They can even accelerate or slow down the process of adaptation. Put all these effects together, and these neuromodulators can tune the neural activity across the entire brain. It is the balance of these different neuromodulators that determines a nematode’s affective state.

The dopamine neurons of nematodes extend small appendages out of its head and contain receptors specifically designed to detect food. When these neurons detect the presence of food, they flood

the brain with dopamine. This tunes circuits to generate

the state of exploitation. This effect can last for minutes before dopamine levels drop again and the nematode returns to the state of escape. Serotonin neurons in nematodes have receptors that detect the presence of

food in their throats, and if enough serotonin is

released, it triggers satiety.

The simple brain of the nematode offers a window into the first, or at least very early, functions of dopamine and serotonin. In the nematode, dopamine is released when food is detected around the worm, whereas serotonin is released when food is detected inside the worm. If dopamine is the something-good-is-nearby chemical, then serotonin is the something-good-is-actually-happening chemical. Dopamine drives the hunt for food; serotonin drives the enjoyment of it once it is being eaten.

While the exact functions of dopamine and serotonin have been elaborated throughout different evolutionary lineages, this basic dichotomy between dopamine and serotonin has been remarkably conserved since the first bilaterians. In species as divergent as nematodes, slugs, fish, rats, and humans, dopamine is released by nearby rewards and triggers the affective state of arousal and pursuit (exploitation); and serotonin is released by the consumption of rewards and triggers a state of low arousal, inhibiting the

pursuit of rewards (satiation). What happens when you see something you want, like food when you’re hungry, a sexy mate, the finish line at the end of a race? In all cases, your brain releases a burst of dopamine. What happens when you get something you want, like when you’re orgasming, eating delicious food, or just finishing a task on your to-do list? Your

brain releases serotonin.

If you raise dopamine levels in the brain of a rat, they begin impulsively exploiting any nearby reward they can find: gorging on food and trying to mate

with whomever they see. If instead you raise their serotonin levels, they stop eating and become less impulsive and more

willing to delay gratification. Serotonin shifts behavior from a focused pursuit of goals to a contented satiety by turning off dopamine responses and by

dulling the responses of valence neurons.

And crucially, all these neuromodulatory neurons—like valence neurons—are also sensitive to internal states. Dopamine neurons are more likely to respond to food cues when an animal is hungry.

This connection between dopamine and reward has caused dopamine to be—incorrectly—labeled the “pleasure chemical.” Kent Berridge, a neuroscientist at the University of Michigan, came up with a experimental paradigm to explore the relationship between dopamine and pleasure. Rats, like humans, make distinct facial expressions when tasting things they like, such as yummy sugar pellets, and things they don’t like, such as bitter liquid. A baby will smile when tasting warm milk and spit when tasting bitter water; rats will lick their lips when they taste yummy food and gape their mouths and shake their heads when they taste gross food. Berridge realized he could use the frequency of these different facial reactions as a proxy for identifying pleasure in rats.

To the surprise of many, Berridge found that increasing dopamine levels in the brains of rats had no impact on the degree and frequency of their pleasurable facial expressions to food. While dopamine will cause rats to consume ridiculous amounts of food, the rats do not indicate they are doing so because they like the food more. Rats do not express a higher number of pleasurable lip smacks. If anything, they express more disgust with the food, despite eating more of it. It is as if rats can’t stop eating even though they no longer enjoy it.

In another experiment, Berridge destroyed the dopamine neurons of several rats, depleting almost all the dopamine in their brains. These rats would sit next to an abundance of

food and starve to death. But this dopamine depletion had no impact on pleasure; if Berridge placed food into the mouths of these hungry rats, they exhibited all the facial expressions suggesting the kind of euphoria one would feel from eating when hungry; they smacked their lips more than ever. Rats experienced pleasure just fine without dopamine—they just didn’t seem motivated to pursue it.

This finding has also been confirmed in humans. In a controversial set of experiments in the 1960s, the psychiatrist Robert Heath implanted electrodes in the brains of humans so that patients could push a button to stimulate their own dopamine neurons. Patients quickly began repeatedly pressing this button, often hundreds

of times an hour. One might assume this was because they “liked” it, but in Heath’s words:

The patient, in explaining why he pressed the septal button with such frequency, stated that the feeling was . . . as if he were building up to a sexual orgasm. He reported that he was unable to achieve the orgastic end point, however, explaining that his frequent, sometimes frantic, pushing of the button was an attempt to

reach the end point.

Dopamine is not a signal for pleasure itself; it is a signal for the anticipation of future pleasure. Heath’s patients weren’t experiencing pleasure; to the contrary, they often became extremely frustrated at their inability to satisfy the incredible cravings the button produced.

Berridge proved that dopamine is less about liking things and more about wanting things. This discovery makes sense given the evolutionary origin of dopamine. In nematodes, dopamine is released when they are near food but not when they are consuming food. The dopamine-triggered behavioral state of exploitation in nematodes—in which they slow down and search their surroundings for food—is in many ways the most primitive version of wanting. As early as the first bilaterians, dopamine was a signal for the anticipation of a future good thing, not the signal for the good thing itself.

While dopamine has no impact on liking reactions, serotonin decreases

both liking and disliking reactions. When given drugs that increase serotonin levels, rats smack their lips less to good food and shake their heads less to bitter food. This is also what we would expect given the evolutionary origin of serotonin: serotonin is the satiation, things-are-okay-now, satisfaction chemical, designed to turn off valence responses.

Dopamine and serotonin are primarily involved in navigating the happy side of affective states—the different flavors of positive affect. There are additional neuromodulators, equally ancient, that undergird the mechanisms of negative affect—of stress, anxiety, and depression.

When Worms Get Stressed

Humanity suffers from stress-related diseases more than ever. More people in the world die from suicide each year than from all violent crimes and wars put together. Around 800,000 people take their lives each year, and there are over

15 million annual suicide attempts. Over 300 million people in the world suffer from depression—stripped of their ability to experience pleasure

and engage in life. Over 250 million people in the world suffer from anxiety disorders—irrationally terrified of the world around them. The Centers for Disease Control (CDC) has even devised a term for this: deaths of despair. The rate of deaths of despair has more than doubled in the last twenty years.

These people aren’t being eaten by lions, or starving, or freezing to death. These people are dying because their brains are killing them. Choosing to commit suicide, knowingly consuming deadly drugs, or binge eating oneself into obesity are, of course, behaviors generated by our brains. Any attempt to understand animal behavior, brains, and intelligence itself is wholly incomplete without understanding this enigma: Why would evolution have created brains with such a catastrophic and seemingly ridiculous flaw? The point of brains, as with all evolutionary adaptations, is to improve survival. Why, then, do brains generate such obviously self-destructive behaviors?

sleep, reproduction, and digestion. These neuromodulators work in part by directly counteracting the effectiveness of serotonin—reducing the ability of an animal to

rest and be content.

Moving around in the world, even for nematodes, consumes a large amount of energy. The adrenaline-induced escape response is one of the most expensive behavioral choices an animal can make—the escape response requires a large expenditure of energy on muscles for rapid swimming. So evolution came up with a trick to save energy and thereby allow the escape response to last longer. Adrenaline not only triggers the behavioral repertoire of escape; it also turns off a swath of energy-consuming activities to divert

energetic resources to muscles. Sugar is expelled from cells across the body, cell growth processes are halted, digestion is paused, reproductive processes are turned off, and the immune system is tamed. This is called the acute stress response—what bodies do immediately in response to negative-valence stimuli.

But like a government running a budget deficit to finance a war, the acute stress response’s postponement of essential bodily functions

cannot go on indefinitely. Therefore, our bilaterian ancestor evolved a counterregulatory response to stress—a whole suite of anti-stress chemicals that prepare the body for the end of the war. One of these anti-stress chemicals was opioids.

Poppies are not the only source of opioids; brains make their own opioids and release them

in response to stressors. When the stressor goes away and adrenaline levels drop, nematodes do not go back to their baseline state. Instead, the leftover anti-stress chemicals initiate a suite of recovery-related processes—immune responses, appetite, and digestion are turned back on. These relief-and-recover chemicals like opioids do this in part by enhancing serotonin and dopamine signals (both of which are

inhibited by acute stressors). Opioids also inhibit negative-valence neurons, which helps an animal recover and rest despite any injuries. This, of course, is why opioids are such potent painkillers across all bilaterians. Opioids also keep certain luxury functions, such as reproductive activities, turned off until the relief-and-recover process is done; this is why opioids decrease sex drive. It is no surprise, then, that nematodes, other invertebrates, and humans all have similar responses to opioids—

prolonged bouts of feeding,

inhibited pain responses,

and inhibited reproductive behavior.

This relief-and-recover state doesn’t merely turn appetite back on; a nematode starved for only twelve hours will eat thirty times more food than

their normally hungry peers. In other words, stress makes nematodes binge food. After binging, these previously starved nematodes “pass out,” spending ten times longer in an immobile state than unstarved worms. Nematodes do this because stress is a signal that circumstances are dire and food may be, or may soon become, scarce. Thus, nematodes stock up on as much food as they can in preparation for the next experience of starvation. As far back as the first brains six hundred million years ago, the system for binge eating after a stressful experience was already put in place.

These anti-stress hormones like opioids differed from dopamine and serotonin in Kent Berridge’s rat facial expression experiments. While dopamine had no impact on liking reactions, giving opioids to a rat did, in fact, substantially increase their liking reactions to food. This makes sense given what we now know about the evolutionary origin of opioids. Opioids are the relief-and-recover chemical after experiencing stress: stress hormones turn positive-valence responses off (decreasing liking), but when a stressor is gone, the leftover opioids turn these valence responses back on (increasing liking). Opioids make everything better; they increase liking reactions and decrease disliking reactions; increasing pleasure and inhibiting pain.

The Blahs and Blues

All this describes what bodies do in response to short-term stressors—the acute stress response. But most of the ways that stress plagues modern humanity comes from what happens to bodies in response to prolonged stressors—the chronic stress response. This, too, has origins in the first bilaterians. If a nematode is exposed to thirty minutes of a negative stimulus (such as dangerous heat, freezing cold, or toxic chemicals), at first it will exhibit the hallmarks of the acute stress response—it will try to escape, and stress hormones will

pause bodily functions. But after just two minutes of no relief from this inescapable stressor, nematodes do something surprising:

they give up. The worm stops moving; it stops trying to escape and just lies there. This surprising behavior is, in fact, quite clever: spending energy escaping is worth the cost only if the stimulus is in fact escapable. Otherwise, the worm is more likely to survive if it conserves energy by waiting. Evolution embedded an ancient biochemical failsafe to ensure that an organism did not waste energy trying to escape something that was inescapable; this failsafe was the early seed of chronic stress and depression.

Any consistent, inescapable, or repeating negative stimuli, such as constant pain or prolonged starvation, will shift a nematode brain into a

state of chronic stress. Chronic stress isn’t all that different from acute stress; stress hormones and opioids remain elevated, chronically inhibiting digestion, immune

response, appetite, and reproduction. But chronic stress differs from acute stress in at least one important way: it turns off arousal and motivation.

The exact biochemical mechanisms of chronic stress, even in nematodes, are complex and not fully understood. But one thing that does seem to be different between the states of acute stress and chronic stress is that chronic

stress starts activating serotonin. At first glance, this makes no sense: serotonin was supposed to be the satiation and good-feels chemical. But consider the main effect of serotonin: it turns off valence neuron responses and lowers arousal. If you add this to the soup of stress hormones, you get a bizarre yet unfortunately familiar state—numbness. This is, perhaps, the most primitive form of depression. Of course, nematodes don’t go through artistic blue periods the way that Picasso did, nor do they necessarily consciously “experience” anything, but nematodes still do share the fundamental feature found in depressive episodes across all bilaterians from insects to fish, to mice, to humans: the numbing of valence responses. This dulls pain and renders even the most exciting stimuli entirely unmotivating. Psychologists call this canonical symptom of depression anhedonia—the lack (an) of pleasure (hedonia).

Anhedonia in animals like nematodes seems to be a trick to preserve energy in the

presence of inescapable stressors. Animals no longer respond to stressors, good food smells, or nearby mates. In humans, this ancient system robs its sufferers of the ability to experience pleasure and motivation. This is the blah or blues of depression. And like all affective states, chronic stress persists after the negative

stimuli have gone away. Such learned helplessness, where animals stop trying to escape from negatively valenced stimuli, is seen even in many bilaterians, including cockroaches,

slugs, and fruit flies.

We have invented drugs that hack these ancient systems. The euphoria provided by natural opioids is meant to be reserved for that brief period after a near-death experience. But humans can now indiscriminately trigger this state with nothing more than a pill. This creates a problem. Repeatedly flooding the brain with opioids creates a state of chronic stress when the drug wears off—adaptation is unavoidable. This then traps opioid users in a vicious cycle of relief, adaptation, chronic stress requiring more drugs to get back to baseline, which causes more adaptation and thereby more chronic stress. Evolutionary constraints cast a long shadow on modern humanity.

These primitive affective states were passed down and elaborated throughout evolution, and remnants are still—whether we like it or not—essential cornerstones of human behavior. Over time, neuromodulators were repurposed for different functions, and new variants of each of these affective states emerged. And so, while the modern emotional states of humans are undeniably more complex and nuanced than a simple two-by-two grid of valence and arousal, they nonetheless retain the scaffolding of the basic template from which they evolved.

Although these affective states are shared across bilaterians, our more distant animal cousins—anemones, coral, and jellyfish—do not show such states. Many of these animals

don’t even have serotonin neurons at all.

This leaves us at the doorstep of a surprising hypothesis: Affect, despite all its modern color, evolved 550 million years ago in early bilaterians for nothing more than the mundane purpose of steering. The basic template of affect seems to have emerged from two fundamental questions in steering. The first was the arousal question: Do I want to expend energy moving or not? The second was the valence question: Do I want to stay in this location or leave this location? The release of specific neuromodulators enforced specific answers to each of these questions. And these global signals for stay and leave could then be used to modulate suites of reflexes, such as whether it was safe to lay eggs, mate, and expend energy digesting food.

However, these affective states and their neuromodulators would go on to play an even more foundational role in the evolution of the first brains.

it we are nothing.

—ERIC KANDEL

ON DECEMBER 12, 1904, a Russian scientist by the name of Ivan Pavlov stood in front of an assembly of researchers at the Karolinska Institute in Sweden. Pavlov had, two days earlier, become the first Russian to win the Nobel Prize. Eight years prior, Alfred Nobel—the Swedish engineer and businessman who got rich from his invention of dynamite—had passed away and bequeathed his fortune to the creation of the Nobel Foundation. Nobel had stipulated that winners were to give a lecture on the subject for which the prize had been awarded, and so, on this day in Stockholm, Pavlov gave his lecture.

Although he is currently known for his contributions to psychology, that was not the work that earned him the Nobel. Pavlov was not a psychologist but a physiologist—he had spent his entire research career up to this point studying the underlying biological mechanisms—the “physiology”—of the digestive system.

Before Pavlov, the only way to study the digestive system was to surgically remove animals’ organs—esophagus, stomach, or pancreas—and run experiments quickly before the organs died. Pavlov pioneered a variety of relatively noninvasive techniques that enabled him to measure features of the digestive system in intact and healthy dogs. The most famous of these was the insertion of a small salivary fistula that diverted saliva from one salivary gland to a small tube that hung out of the dog’s mouth; this enabled Pavlov to determine the quantity and content of saliva produced by various stimuli. He did similar tricks with the esophagus, stomach, and pancreas.

Through these new techniques, Pavlov and his colleagues made several discoveries. They learned what types of digestive chemicals were released in response to various foods, and he discovered that the digestive organs were under the control of the nervous system. These contributions won him the prize.

However, two-thirds through his speech, Pavlov turned his focus away from his prizewinning work. An excitable scientist, he couldn’t resist pitching research that was, at the time, speculative but that he believed would eventually become his most important work—his exploration of what he called conditional reflexes.

There had always been a pesky confound that got in the way of his meticulous measurements of digestive responses—digestive organs often became stimulated before animals tasted food. His dogs salivated and their stomachs gurgled the moment they realized an experiment with food was about to begin. This was a problem. If you want to measure how salivary glands respond when taste buds detect fatty meat or sugary fruit, you don’t want the confounding measurement of whatever was released by the subjects merely looking at these substances.

This psychic stimulation, as it was called, was a particular annoyance to Pavlov’s research, what he deemed a

“source of error.” Pavlov developed various techniques to eliminate this confound; for instance, experimenters worked in separate isolated rooms to “carefully avoid everything that could

elicit in the dog thoughts about food.”

Only much later, after bringing psychologists into his lab, did Pavlov begin to view psychic stimulation not as a confound to be eliminated but as a variable worthy of analysis. Ironically, it was a digestive physiologist with the goal of eliminating psychic stimulation who became the first to understand it.

Pavlov’s lab discovered that psychic stimulation was not as random as it seemed. Dogs would salivate in response to any stimuli—metronomes, lights, buzzers—that had been previously associated with food. If an experimenter turned on a buzzer and then gave food, the dog began to salivate in response to the buzzer alone. The dog had developed a conditional reflex—the reflex to salivate in response to the buzzer was conditional on the prior association between the buzzer and food. Pavlov contrasted these conditional reflexes with what he called unconditional reflexes—those that were innate and required no association. A hungry dog’s reflex to salivate in response to sugar placed in its mouth occurred regardless of any prior associations.

Shortly after Pavlov’s experiments, other scientists began trying these techniques on other reflexes. It turned out that most, if not all, reflexes build such associations. Pair an arbitrary sound with an electric shock to your hand, and soon your hand will retract to just the sound. Pair an arbitrary sound with a gentle puff of air to someone’s eyes, and eventually he or she will involuntarily blink in response to just the sound. Pair an arbitrary sound with a reflex hammer to a person’s knee, and eventually their leg will

kick in response to just the sound.

The defining feature of Pavlov’s conditional reflexes is that they are involuntary associations; people can’t help but blink, kick, or retract their hands. Just as a soldier who returns from war cannot help but jump when they hear a loud noise or a person with a phobia of public speaking cannot help but tense up before going onstage, so too could Pavlov’s dogs not help but salivate in response to the buzzer. The involuntary nature of Pavlov’s conditional reflexes, the fact that associative learning occurs automatically without conscious involvement, was the first clue that learning and memory might be more ancient than previously thought. Learning might not require all the brain structures that emerged later in evolution. Indeed, even a rat with its entire brain removed exhibits conditional reflexes. If you pair a tap to its leg (which causes leg retraction) with a tap to its tail (which causes tail retraction), then it will learn to retract its tail in response to a leg tap, despite lacking a brain. Rats can learn this association with nothing but the

simple circuits in their spinal cords.

If associative learning is a property of simple circuits of neurons, even those present outside of the brain, then it might be a very ancient evolutionary trick. Indeed, Pavlov had unintentionally stumbled on the evolutionary origin of learning itself.

Tweaking the Goodness and Badness of Things

Suppose you took a hundred nematodes, put half of them in a dish with plain water and the other half in a dish with salty water. After several hours, these nematodes will become uncomfortably hungry, as neither dish contains any food. At this point, put both groups of nematodes into another dish that contains a little morsel of salt on one side. What happens?

The nematodes that experienced hunger with plain water will behave as normal nematodes do: they will steer toward the salt (nematodes typically consider salt to have positive valence). However, the nematodes that experienced hunger in salt water will do the exact opposite: they will steer

away from the salt.

from that food smell. If you feed nematodes at a specific temperature, they will shift their preferences

toward that temperature. Pair a gentle tap to the side of a slug with a small electric shock, which triggers a withdrawal reflex, and the slug will learn to withdraw to just the tap, an association that

will last for days.

And yet, while associative learning is found across bilaterians, our most distant animal cousins—the radially symmetric jellyfish, anemones, and coral—are not

capable of learning associations.

a process called extinction.

There are two interesting features of extinction. Suppose you break a previously learned association—sound the buzzer several times in a row, but don’t give food. As expected, dogs will eventually stop salivating at the buzzer. However, if you wait a few days and then sound the buzzer again, something odd happens: dogs start salivating in response to the buzzer again. This is called spontaneous recovery: broken associations are rapidly suppressed but not, in fact, unlearned; given enough time, they reemerge. Further, if after a long stretch of trials with a broken association (buzzer but no food), you reinstate the association (sound a buzzer and provide food again), the old association will be relearned far more rapidly than the first time the dog experienced the association between the buzzer and food. This is called reacquisition: old extinguished associations are reacquired faster than entirely new associations.

Why do associations show spontaneous recovery and reacquisition? Consider the ancient environment in which associative learning evolved. Suppose a worm has many experiences of finding food alongside salt. And then one day, it detects salt, steers toward it, and finds no food. After the worm spends an hour sniffing around without finding food, the association becomes extinguished, and the worm begins steering toward other cues, no longer attracted to salt. If two days later it detects salt again, would it be smarter to steer toward or away from it? In all of the worm’s past experiences, except the most recent one, when it smelled salt it also found food. And so the smarter choice would be to steer toward salt again—the most recent experience may have been a fluke. This is the benefit of spontaneous recovery—it enables a primitive form of long-term memory to persist through the tumult of short-term changes in the contingencies of the world. Of course, if the next twenty times the worm detects salt it fails to find food, the association may eventually be permanently extinguished.

The effect of reacqusition—the accelerated relearning of old previously broken associations—evolved in ancient worms for similar reasons. Suppose this same worm finds salt alongside food after the association was long ago extinguished. How quickly should the worm restrengthen the association between salt and food? It would make sense to relearn this association rapidly, given the long-term memory the worm has: In some cases, salt leads to food, and it seems that right now is one of those situations! Thus, old associations are primed to reemerge whenever the world provides hints that old contingencies are newly reestablished.

Spontaneous recovery and reacquisition enabled simple steering brains to navigate changing associations, temporarily suppress old associations that were currently inaccurate, and remember and relearn broken associations that became effective again.

The first bilaterians used these tricks of acquisition, extinction, spontaneous recovery, and reacquisition to navigate changing contingencies in their world. These solutions to continual learning are found across many reflexes in many animals, even the most ancient animals like the nematode, embedded in the simplest neural circuits, inherited from the first brain, originally crafted to make steering work in the ever-changing world of the

ancient Ediacaran Sea.

The Credit Assignment Problem

Associative learning comes with another problem: When an animal gets food, there is never a single predictive cue beforehand but rather a whole swath of cues. If you pair a tap to the side of a slug with a shock, how does a slug’s brain know to associate only the tap with the shock and not the many other sensory stimuli that were present, such as the surrounding temperature, the texture of the ground, or the diverse chemicals floating around the seawater? In machine learning, this is called the credit assignment problem: When something happens, what previous cue do you give credit for predicting it? The ancient bilaterian brain, which was capable of only the simplest forms of learning, employed four tricks to solve the credit assignment problem. These tricks were both crude and clever, and they became foundational mechanisms for how neurons make associations in all their bilaterian descendants.

The first trick used what are called eligibility traces. A slug will associate a tap with a subsequent shock only if the tap occurs one second before the shock. If the tap occurs two seconds or more before the shock, no association will be made. A stimulus like a tap creates a short eligibility trace that lasts for about a second. Only within this short time window can associations be made. This is clever, as it invokes a reasonable rule of thumb: stimuli that are useful for predicting things should occur right before the thing you are trying to predict.

The second trick was overshadowing. When animals have multiple predictive cues to use, their brains tend to pick the cues that are the strongest—strong cues overshadow weak cues. If a bright light and a weak odor are both present before an event, the bright light, not the weak odor, will be used as the predictive cue.

The third trick was latent inhibition—stimuli that animals regularly experienced in the past are inhibited from making future associations. In other words, frequent stimuli are flagged as irrelevant background noise. Latent inhibition is a clever way to ask, “What was different this time?” If a slug has experienced the current texture of the ground and the current temperature a thousand times but has never experienced a tap before, then the tap is far more likely to be used as a predictive cue.

The fourth and final trick for navigating the

credit assignment problem was blocking. Once an animal has established an association between a predictive cue and a response, all further cues that overlap with the predictive cue are blocked from association with that response. If a slug has learned that a tap leads to shock, then a new texture, temperature, or chemical will be blocked from being associated with the shock. Blocking is a way to stick to one predictive cue and avoid redundant associations.

The Original Four Tricks for Tackling the Credit Assignment Problem

ELIGIBILITY TRACES	OVERSHADOWING	LATENT INHIBITION	BLOCKING
Pick the predictive cue that occurred between 0 to 1 second before the event.	Pick the predictive cue that was the strongest.	Pick the predictive cue that you haven’t seen before.	Stick to predictive cues once you have them and ignore others.

Eligibility traces, overshadowing, latent inhibition, and blocking are ubiquitous across Bilateria. Pavlov identified these in the conditional reflexes of his salivating dogs; they are found in the involuntary reflexes of humans; and they are seen in the associative learning of flatworms, nematodes, slugs, fish, lizards, birds, rats, and most every bilaterian

in the animal kingdom. These tricks for navigating the credit assignment problem evolved as far back as the very first brains to make associative learning work.

These tricks are hardly perfect. In some circumstances, the best predictive cue may occur one minute before the event, not one second before. In other circumstances, the best predictive cue may be the weak cue, not the strong one. Over time, brains evolved more sophisticated strategies for solving the credit assignment problem (stay tuned for

blocking, and overshadowing. Along with acquisition, extinction, spontaneous recovery, and reacquisition, this portfolio of tricks make up the foundation of the neural mechanisms of associative learning, mechanisms that are embedded deep into the inner workings of neurons, neural circuits, and brains themselves.

The Ancient Mechanisms of Learning

For thousands of years, two groups of philosophers have been debating the relationship between the brain and the mind. One group, the dualists, like Plato, Aquinas, and Descartes, argue that the mind exists separately from the brain. The entities might interact with each other, but they are distinct; the mind is something beyond the physical. The materialists, like Kanada, Democritus, Epicurus, and Hobbes, argued that whatever the mind is, it is located entirely in the physical structure of the brain. There is nothing beyond the physical. This debate still rages in philosophy departments around the world. If you have made it this far in the book, I will assume you lean on the side of materialism, that you—like me—tend to reject nonphysical explanations for things, even the mind. But by siding with the materialists, we introduce several issues that, at first, are hard to explain physically, the most obvious being learning.

You can read a sentence once and then immediately repeat it out loud. If we stick to a materialist view, this means that reading this sentence instantaneously changed something physical in your brain. Anything that leads to learning causes physical reorganization of something in the 86 billion neurons in each of our heads. Keeping track of a conversation, watching a movie, and learning to tie your shoes all must change the physicality of our brains.

People have been speculating on the physical mechanisms of learning for thousands of years, and even the dualists pontificated on materialist explanations for learning. Plato believed that the brain was like a wax tablet in which perceptions left lasting impressions; he believed that

memories were these impressions. Descartes argued that memories were formed through creating new “folds” in the brain, “not unlike the folds which remain in this paper after it

has once been folded.” Others speculated that memories

were persistent “vibrations.” These ideas were all wrong, although through no fault of their originators; at the time, no one understood even the basic building blocks of the nervous system, so they could not even begin to conceive of

how learning might work.

The flurry of discoveries about neurons in the early twentieth century provided a host of new building blocks. The discovery of the connections between neurons—synapses—was the most obvious new thing that could presumably change in the brain during learning. Indeed, it turns out that learning rises not from impressions, folds, or vibrations but from changes to these synaptic connections.

Learning occurs when synapses change their strength or when new synapses are formed or old synapses are removed. If the connection between two neurons is weak, the input neuron will have to fire many spikes to get the output neuron to spike. If the connection is strong, the input neuron will have to fire only a few spikes to get the output neuron to spike. Synapses can increase their strength by the input neuron releasing more neurotransmitter in response to a spike or the postsynaptic neuron increasing the number of protein receptors (hence more responsive to the same quantity of neurotransmitter).

Synapses have many mechanisms for choosing when to strengthen or weaken. These mechanisms are extremely old evolutionary innovations, originating from the associative learning of the first bilaterians. For example, there is clever protein machinery in the synapses of bilaterian neurons that detect whether the input neuron fired within a similar time window as the output neuron. In other words, individual connections can detect whether an input (like the sensory neuron activated by a tap) is activated at the same time as an output (like the motor neuron activated by a shock). And if these neurons are activated at the same time, this protein machinery triggers a process that strengthens the synapse.

Figure 4.3

Figure by Max Bennett

involved in Hebbian learning. But given our evolutionary past, this is to be expected: if our shared ancestor with coral polyps had no associative learning, then we should expect them to lack the mechanisms that underly such learning.

Learning had humble beginnings. While early bilaterians were the first to learn associations, they were still unable to learn most things. They could not learn to associate events separated by more than a few seconds; they could not learn to predict the exact timing of things; they could not learn to recognize objects; they could not recognize patterns in the world; and they could not learn to recognize locations or directions.

But still, the ability of the human brain to rewire itself, to make associations between things, is not a uniquely human superpower but one we inherited from this ancient bilaterian ancestor that lived over 550 million years ago. All the feats of learning that followed (the ability to learn spatial maps, language, object recognition, music, and everything else) were built on these same learning mechanisms. From the bilaterian brain onward, the evolution of learning was primarily a process of finding new applications of preexisting synaptic learning mechanisms, without changing the learning mechanisms themselves.

Learning was not the core function of the first brain; it was merely a feature, a trick to optimize steering decisions. Association, prediction, and learning emerged for tweaking the goodness and badness of things. In some sense, the evolutionary story that will follow is one of learning being transformed from a cute feature of the brain to its core function. Indeed, the next breakthrough in brain evolution was all about a brilliant new form of learning, one that was possible only because it was built on the foundation of valence, affect, and associative learning.

Figure 5.2: Our Cambrian ancestors

Original art by Rebecca Gelernter

are incredibly similar.

The first animals gifted us neurons. Then early bilaterians gifted us brains, clustering these neurons into centralized circuits, wiring up the first system for valence, affect, and association. But it was early vertebrates who transformed this simple proto-brain of early bilaterians into a true machine, one with subunits, layers, and processing systems.

The question is, of course, what did this early vertebrate brain do?

Thorndike’s Chickens

Around the same time that Ivan Pavlov was unraveling the inner workings of conditional reflexes in Russia, an American psychologist by the name of Edward Thorndike was probing animal learning from a different perspective.

In 1896, Edward Thorndike found himself in a room full of chickens. Thorndike had recently enrolled in Harvard’s master’s program in psychology. His main interest was studying how children learn: How best can we teach children new things? He had numerous ideas for experiments, but to Thorndike’s chagrin, Harvard would not allow him to conduct experiments on human children. So Thorndike had no choice but to focus on subjects that were easier to obtain: chickens, cats, and dogs.

To Thorndike, this wasn’t all bad. A staunch Darwinist, he was unwavering in his view that there should be common principles in the learning of chickens, cats, dogs, and humans. If these animals shared a common ancestor, then they all should have inherited similar learning mechanisms. By probing how these other animals learned, he believed he might be able to also illuminate the principles of how humans learned.

Thorndike was both extremely shy and incredibly smart, so he was perhaps the perfect person to engage in the solitary, meticulously repetitive, and undeniably clever animal studies that he pioneered. Pavlov did his groundbreaking psychology work when he was middle-aged, after an already famed career as a physiologist, but Thorndike’s most famous work was his first. It was his doctoral dissertation, published in 1898, when he was twenty-three, for which he is most well known. His dissertation: “Animal Intelligence: An Experimental Study of the Associative Processes in Animals.”

Thorndike’s genius, like Pavlov’s, was in how he reduced hopelessly complex theoretical problems to simple measurable experiments. Pavlov explored learning by measuring the amount of saliva released in response to a buzzer. Thorndike explored learning by measuring the speed with which animals learned to escape from what he called puzzle boxes.

Thorndike constructed a multitude of cages, each with a different puzzle inside that, if solved correctly, would open an escape door. These puzzles weren’t particularly complex—some had latches that when pushed would open the door; others had hidden buttons; others had hoops to pull. Sometimes the puzzle did not require a physical contraption, and Thorndike would just manually open the door whenever the animal did something specific, such as lick itself. He placed various animals in these cages, put food outside to motivate the animals to get out of the boxes, and measured exactly how long it took them to figure out the puzzle.

Once the animal escaped, he would record the animal’s time, and then have the animal do it again, and again, and again. He would calculate the average time it took animals to solve a given puzzle on their first trial, compare that with the time for their second, and go all the way to how fast they solved it after as many as a hundred trials.

Thorndike originally wanted to probe the dynamics of imitation, a feature of learning he believed would exist across

many animal species. He allowed untrained cats to watch trained cats escape from various puzzle boxes to see if it had any effect on their own learning. In other words, could cats learn through imitation? It seemed at the time that the answer was no; they didn’t get any better by watching (note that some animals can do this; stay tuned for

When first placed in a cage, the cat would try a whole host of behaviors: scratching at the bars, pushing at the ceiling, digging at the door, howling, trying to squeeze through the bars, pacing around the cage. Eventually the cat would accidently press the button or pull the hoop, and the door would open; the cat would exit and happily eat its prize. The animals became progressively faster at repeating the behaviors that got them out of the box. After many trials, cats stopped doing any of their original behaviors and immediately performed the actions required to escape. These cats were learning through trial and error. He could quantify this trial-and-error learning with the gradual decay in the time it took for animals to escape (f igure 5.6).

What was most surprising was how much intelligent behavior emerged from something as simple as trial-and-error learning. After enough trials, these animals could effortlessly perform incredibly complex sequences of actions. It was originally believed that the only way to explain such intelligent behavior in animals was through some notion of insight or imitation or planning, but Thorndike showed how simple trial and error was all an animal really needed. Thorndike summarized his result in his now famous law of effect:

Responses that produce a satisfying effect in a particular situation become more likely to occur

again in that situation, and responses that produce a discomforting effect become less likely to occur again in that situation.

Animals learn by first performing random exploratory actions and then adjusting future actions based on valence outcomes—positive valence reinforces recently performed actions, and negative valence un-reinforces previously performed actions. The terms satisfying and discomforting went out of favor over the decades following Thorndike’s original research; they had an uncomfortable allusion to an actual internal sensation or feeling. Psychologists, including Thorndike, eventually replaced the terms satisfying and discomforting with reinforcing and punishing.

One of Thorndike’s intellectual successors, B. F. Skinner, went so far as to suggest that all animal behavior, even in humans, was a consequence of nothing more than trial and error. As we will see with

specific button to get food; fish can learn to swim through a small escape hatch to avoid

getting caught in a net; and fish can even learn to jump through hoops to get food. Fish can remember how to do these tasks for months or even years after being trained. The process of learning is the same in all these tests: fish try some relatively random actions and then progressively refine their behavior depending on what gets reinforced. Indeed, Thorndike’s trial and error learning often goes by another name: reinforcement learning.

If you tried to teach a simple bilaterian like a nematode, flatworm, or slug to perform any of these tasks, it would fail. A nematode cannot be trained to perform arbitrary sequences of actions; it will never learn to navigate through hoops to get food.

Over the next four chapters we will explore the challenges of reinforcement learning and learn why ancestral bilaterians, like modern nematodes, were unable to learn this way. We will learn about how the first vertebrate brains worked, how they overcame these earlier challenges, and how these brains flowered into general reinforcement learning machines.

The second breakthrough was reinforcement learning: the ability to learn arbitrary sequences of actions through trial and error. Thorndike’s idea of trial-and-error learning sounds so simple—reinforce behaviors that lead to good things and punish behaviors that lead to bad things. But this is an example where our intuitions about what is intellectually easy and what is hard are mistaken. It was only when scientists tried to get AI systems to learn through reinforcement that they realized that it wasn’t as easy as Thorndike had thought.

not going to work.

Here’s why. Suppose we teach an AI to play checkers using Thorndike’s version of trial-and-error learning. This AI would start by making random moves, and we would give it a reward whenever it won and a punishment whenever it lost. Presumably, if it played enough games of checkers, it should get better. But here’s the problem: The reinforcements and punishments in a game of checkers—the outcome of winning or losing—occur only at the end of the game. A game can consist of hundreds of moves. If you win, which moves should get credit for being good? If you lose, which moves should get credit for being bad?

This is, of course, just another version of the credit assignment problem we saw in

behaviors using predicted rewards? Put another way: Instead of rewarding an AI system when it wins, what if you reward it when the AI system thinks it is winning?

Sutton decomposed reinforcement learning into two separate components: an actor and a critic. The critic predicts the likelihood of winning at every moment during the game; it predicts which board configurations are great and which are bad. The actor, on the other hand, chooses what action to take and gets rewarded not at the end of the game but whenever the critic thinks that the actor’s move increased the likelihood of winning. The signal on which the actor learns is not rewards, per se, but the temporal difference in the predicted reward from one moment in time to the next. Hence Sutton’s name for his method:

Imagine you are playing checkers. For the first nine moves, it is mostly neck and neck between you and your opponent. And then on the tenth move you pull off some clever maneuver that turns the tide of the game; suddenly you realize you are in a far better position than your opponent. It is that moment where a temporal difference learning signal reinforces your action.

This, Sutton proposed, might solve the temporal credit assignment problem. This would enable an AI system to learn as it goes instead of having to wait until the end of each game. An AI system can reinforce some moves and punish others throughout a long game of checkers, whether or not it won or lost the overall game. Indeed, sometimes a player makes many good moves in a game he or she ultimately loses, and sometimes a player makes many bad moves in a game he or she ultimately wins.

Despite the intuitive appeal of Sutton’s approach, we should not expect it to work. Sutton’s logic is circular. The critic’s prediction of how likely you are to win given a board position depends on what future actions the actor will take (a good board position isn’t good if the actor doesn’t know how to take advantage of it). Similarly, the actor’s decision of what action to take depends on how accurate the critic’s temporal difference reinforcement signals have been at reinforcing and punishing past actions. In other words, the critic depends on the actor, and the actor depends on the critic. This strategy seems doomed from the start.

In his simulations, however, Sutton found that by training an actor and a critic simultaneously, a magical bootstrapping occurs between them. Sure, in the beginning the critic often rewards the wrong actions, and the actor often fails to take the necessary actions to fulfill the predictions of the critic. But over time, with enough games, each refines the other until they converge to produce an AI system capable of making remarkably intelligent decisions. At least, that’s what happened in Sutton’s simulations. It wasn’t clear whether this would work in practice.

At the same time that Sutton was working on TD learning, a young physicist by the name of Gerald Tesauro was working on getting AI systems to play backgammon. Tesauro was at IBM Research, the same group that would later build Deep Blue (the program that famously beat Garry Kasparov in chess) and Watson (the program that famously beat Ken Jennings in Jeopardy!). But before Deep Blue or Watson, there was Neurogammon. Neurogammon was a backgammon-playing AI system that was trained on transcripts of hundreds of expertly played backgammon games. It learned not through trial and error but by attempting to replicate what it believed a human expert would do. By 1989, Neurogammon could beat every other backgammon-playing computer program, but it was lackluster compared to a human, unable to beat even an intermediate-level player.

By the time Tesauro stumbled on Sutton’s work on TD learning, he had spent years trying every conceivable technique to get his computer to play backgammon as well as a human. His crowning achievement was Neurogammon, which was clever but stuck at an intermediate level. And so Tesauro was open to new ideas, even Sutton’s radical idea of allowing a system to teach itself from its own predictions.

It was Tesauro who first put Sutton’s idea to a practical test. In the early 1990s he began working on TD-Gammon, a system that learned to play backgammon using temporal difference learning.

Tesauro was skeptical. Neurogammon had been taught with examples from expert human players—it was shown the best moves—while TD-Gammon learned solely from trial and error, requiring it to discover the best moves on its own. And yet, by 1994, TD-Gammon achieved, in Tesauro’s own words, a “truly

staggering level of performance.” It not only blew Neurogammon away but was as good as some of the best backgammon players in the world. While it was Sutton who proved that temporal difference learning worked in theory, it was Tesauro who proved it worked in practice. In the decades that followed, TD learning would be used to train AI systems to do many tasks with human-level skill, from playing Atari games to changing lanes in self-driving cars.

The real question, however, was whether TD learning was merely a clever technique that happened to work or a technique that captured something fundamental about the nature of intelligence. Was TD learning a technological invention, or was it, as Sutton had hoped, an ancient technique that evolution had stumbled upon and long ago weaved into animal brains to make reinforcement learning work?

The Grand Repurposing of Dopamine

While Sutton had hoped there was a connection between his idea and the brain, it was one of his students, Peter Dayan, who found it. At the Salk Institute in San Diego, Dayan and his fellow postdoc Read Montague were convinced that brains implemented some form of TD learning. In the 1990s, emboldened by the success of Tesauro’s TD-Gammon, they went hunting for evidence in the ever-growing mound of neuroscience data.

They knew where to start. Any attempt to understand how reinforcement learning works in vertebrate brains surely had to begin with a little neuromodulator we have already seen: dopamine.

Deep within the midbrain of all vertebrates is a small cluster of dopamine neurons. These neurons, while few in number, send their output to many regions of the brain. In the 1950s researchers discovered that if you put an electrode into the brain of a rat and stimulate these dopamine neurons, you can get a rat to do pretty much anything. If you stimulate these neurons every few times a rat pushes a lever, the rat will push this lever over five thousand times an hour

for twenty-four hours straight. In fact, if given the choice between a dopamine-releasing lever and eating food, rats will choose the lever. Rats will ignore food and starve themselves in

favor of dopamine stimulation.

This effect is also found in fish. A fish will return to places where it is given dopamine and continue doing so even if those areas are paired with unpleasant things it usually avoids (like being

repeatedly removed from water).

Indeed, most drugs of abuse—alcohol, cocaine, nicotine—work by triggering the release of dopamine. All vertebrates, from fish to rats to monkeys to humans, are susceptible to becoming addicted to

such dopamine-enhancing chemicals.

Dopamine was undeniably related to reinforcement, but how exactly was not so clear. The original interpretation was that dopamine was the brain’s pleasure signal; animals repeated behaviors that activated dopamine neurons because it felt good. This made sense in Thorndike’s original concept of trial-and-error learning as a process of repeating behaviors that led to satisfying outcomes. But we already saw in

temporal difference learning signal. Dopamine neurons in Schultz’s monkeys got excited by predictive cues because these cues led to an increase in predicted future rewards (a positive temporal difference); dopamine neurons were unaffected by the delivery of an expected reward because there was no change in predicted future reward (no temporal difference); and dopamine-neuron activity decreased when expected rewards were omitted because there was a decrease in predicted future rewards (a negative temporal difference).

Even the subtleties of dopamine responses aligned exactly with a temporal difference signal. For example, Schultz found that a cue that predicts food in four seconds triggers more dopamine than a cue that predicts

food in sixteen seconds. This is called discounting, something Sutton also incorporated into his TD-learning signal; discounting drives AI systems (or animals) to choose actions that lead to rewards sooner rather than later.

Even the way dopamine responds to probabilities aligned with a TD-learning signal—a cue that predicts food with a 75 percent probability triggers more dopamine than a cue that predicts food with a 25 percent probability.

Dopamine is not a signal for reward but for reinforcement. As Sutton found, reinforcement and reward must be decoupled for reinforcement learning to work. To solve the temporal credit assignment problem, brains must reinforce behaviors based on changes in predicted future rewards, not actual rewards. This is why animals get addicted to dopamine-releasing behaviors despite it not being pleasurable, and this is why dopamine responses quickly shift their activations to the moments when animals predict upcoming reward and away from rewards themselves.

In 1997 Dayan and Montague published a landmark paper, coauthored with Schultz, titled

“A Neural Substrate of Prediction and Reward.” To this day, this discovery represents one of the most famous and beautiful partnerships between AI and neuroscience. A strategy inspired by how Sutton thought the brain might work turned out to successfully overcome practical challenges in AI, and this in turn helped us interpret mysterious data about the brain. Neuroscience informing AI, and AI informing neuroscience.

Most studies that record the activity of dopamine neurons have been done in mammals, but there is every reason to believe these properties of dopamine extend to fish as well. The dopamine-system circuitry is largely the same in both fish and mammal brains, and the same TD-learning signals have been found in the brain structures of fish, rat,

monkey, and human brains. In contrast, no TD-learning signals have been found in the dopamine neurons of nematodes or other simple bilaterians.

for a failed prediction of future reward.

Indeed, you can train vertebrates, even fish, to perform arbitrary actions not only with rewards and punishments but also with the omission of

expected rewards or punishments. To some, a surprise piece of dessert (a reward) is as reinforcing as a surprise day off from school (the omission of something expected but disliked).

A nematode, on the other hand, cannot learn to perform arbitrary behaviors through the omission of rewards. Even crabs and honeybees, who independently evolved many intellectual faculties, are not be able to learn from omission of things.

five seconds after light.

Many forms of life have mechanisms to track the passage of time. Bacteria, animals, and plants all have circadian clocks to track the

cycle of the day. But vertebrates are unique in the precision with which they can measure time. A verterbate can remember that one event occurs precisely five seconds after another event. In contrast, simple bilaterians like slugs and flatworms are entirely unable to learn the precise time

intervals between events. Indeed, simple bilaterians like slugs cannot even learn to associate events separated by more than two seconds, let alone learn that one thing happens exactly five seconds after another. Even the advanced invertebrates like crabs and bees are unable to learn precise time intervals between events.

TD learning, disappointment, relief, and the perception of time are all related. The precise perception of time is a necessary ingredient to learn from omission, to know when to trigger disappointment or relief, and thereby to make TD learning work. Without time perception, a brain cannot know whether something was omitted or simply hasn’t happened yet; our fish would know that the light was associated with a zap but not when it should occur. Our fish would cower in fear in the presence of the light long after the risk of the zap had passed, blind to its own safety. It is only with an inner clock that fish can predict the exact moment the zap would occur and thus, if omitted, the exact moment it deserves a relieving dopamine burst.

The Basal Ganglia

My favorite part of the brain is a structure called the basal ganglia.

For most brain structures, the more one learns about them, the less one understands them—simplified frameworks crumble under the weight of messy complexity, the hallmark of biological systems. But the basal ganglia is different. Its inner wiring reveals a mesmerizing and beautiful design, exposing an orderly computation and function. As one might feel awe that evolution could construct an eye, with such symmetry and elegance, one could equally feel awe that evolution could construct the basal ganglia, also endowed with its own symmetry and elegance.

The basal ganglia is wedged between the cortex and the thalamus (see the figure in the first pages of this book). The input to the basal ganglia comes from the cortex, thalamus, and midbrain, enabling the basal ganglia to monitor an animal’s actions and external environment. Information then flows through a labyrinth of substructures within the basal ganglia, branching and merging, transforming and permuting until it reaches the basal ganglia’s output nucleus, which contains thousands to millions of inhibitory neurons that send massive and powerful connections to motor centers in the brainstem. This output nucleus of the basal ganglia is, by default, activated. The motor circuits of the brainstem are constantly being suppressed and gated by the basal ganglia. It is only when specific neurons in the basal ganglia turn off that specific motor circuits in the brainstem are ungated from activation. The basal ganglia is thereby in a perpetual state of gating and ungating specific actions, operating as a global puppeteer of an animal’s behavior.

The functioning of the basal ganglia is essential to our lives. The cononical symptom of Parkinson’s disease is the inability to initiate movement. Patients will sit in a chair for many minutes before they can muster the will to even sit up. This symptom of Parkinson’s disease primarily emerges due to disruption of the basal ganglia, leaving it in a perpetual state of gating all actions, thereby depriving patients of the ability to initiate even the simplest of movements.

What is the computation performed by the basal ganglia? How does it use incoming information about an animal’s actions and external environment to decide which actions to gate (prevent from occuring) and which actions to ungate (allow to occur)?

In addition to receiving information about an animal’s actions and external environment, the basal ganglia also receives input from a cluster of dopamine neurons. Whenever these dopamine neurons get excited, the basal ganglia is rapidly flooded with dopamine; whenever these dopamine neurons are inhibited, the basal ganglia is rapidly starved of dopamine. The synapses within the basal ganglia have different dopamine receptors, each responding in unique ways; these fluctuating levels of dopamine strengthen and weaken specific synapses, modifying how the basal ganglia processes input.

As neuroscientists traced the circuitry of the basal ganglia, its function became quite clear. The basal ganglia learns to repeat actions

that maximize dopamine release. Through the basal ganglia, actions that lead to dopamine release become more likely to occur (the basal ganglia ungates those actions), and actions that lead to dopmaine inhibition become less likely to occur (the basal ganglia gates those actions). Sound familiar? The basal ganglia is, in part, Sutton’s “actor”—a system designed to repeat behaviors that lead to reinforcement and inhibit behaviors that lead to punishment.

Remarkably, the circuitry of the basal ganglia is practically identical between a human brain and a lamprey fish brain, two species whose shared ancestors were the first vertebrates over 500 million years ago. The various subclusters, the types of neurons, and the overall function seem to be the same. In the brain of early vertebrates emerged the basal ganglia, the biological locus of reinforcement learning.

Reinforcement learning emerged not from the basal ganglia acting alone, but from an ancient interplay between the basal ganglia and another uniquely vertebrate structure called the hypothalamus, which is a small structure at the base of the forebrain.

In vertebrate brains, dopamine release is initially controlled by the hypothalamus. It is the hypothalamus that houses valence neurons inherited from the valence sensory apparatus of ancestral bilaterians. When you are cold, it is your hypothalamus that triggers shivering and that makes you enjoy warmth; just as when you are hot, it is your hypothalamus that triggers sweating and that makes you enjoy the cold. When your body needs calories, it is your hypothalamus that detects hunger signals in your bloodstream and that makes you hungry. The positive valence food-sensitive neurons in early bilaterians functioned just as the positive valence food-sensitive neurons in your hypothalamus do, becoming highly responsive to food when you are hungry and less responsive to food when you are full. This is why you will be salivating over pizza one moment, but then after engorging yourself want absolutely nothing to do with pizza just ten minutes later.

In other words, the hypothalamus is, in principle, just a more sophisticated version of the steering brain of early bilaterians; it reduces external stimuli to good and bad and triggers reflexive responses to each. The valence neurons of the hypothalamus connect to the same cluster of dopamine neurons that propogates dopamine throughout the basal ganglia. When the hypothalamus is happy, it floods the basal ganglia with dopamine, and when it is upset, it deprives the

basal ganglia of dopamine. And so, in some ways, the basal ganglia is a student, always trying to satisfy its vague but stern hypothalamic judge.

The hypothalamus doesn’t get excited by predictive cues; it gets excited only when it actually gets what it wants—food when hungry, warmth when cold. The hypothalamus is the decider of actual rewards; in our AI-playing-backgammon metaphor, the hypothalamus tells the brain whether it won or lost the game but not how well it is doing as the game is unfolding.

But as Minsky found with his attempts to make reinforcement learning algorithms in the 1950s, if brains learned only from actual rewards, they would never be able to do anything all that intelligent. They would suffer from the problem of temporal credit assignment. So then how is dopamine transformed from a valence signal for actual rewards to a temporal difference signal for changes in predicted future reward?

In all vertebrates, there is a mysterious mosaic of parallel circuits within the basal ganglia, one that flows down to motor circuits and gates movement, and another that flows back toward

dopamine neurons directly. One leading theory of basal ganglia function is that these parallel circuits are literally Sutton’s actor-critic system for implementing temporal difference learning. One circuit is the “actor,” learning to repeat the behaviors that trigger dopamine release; the other circuit is the “critic,” learning to predict future rewards and trigger its own dopamine activation.

In our metaphor, the basal ganglian student initially learns solely from the hypothalamic judge, but over time learns to judge itself, knowing when it makes a mistake before the hypothalamus gives any feedback. This is why dopamine neurons initially respond when rewards are delivered, but over time shift their activation toward predictive cues. This is also why receiving a reward that you knew you were going to receive doesn’t trigger dopamine release; predictions from the basal ganglia cancel out the excitement from the hypothalamus.

The beautifully conserved circuitry of the basal ganglia, first emerging in the minuscule brain of early vertebrates and maintained for five hundred million years, seems to be the biological manifestation of Sutton’s actor-critic system. Sutton discovered a trick that evolution had already stumbled upon over five hundred million years ago.

TD learning, the wiring of vertebrate basal ganglia, the properties of dopamine responses, the ability to learn precise time intervals, and the ability to learn from omissions are all interwoven into the same mechanisms for making trial-and-error learning work.

specific types of molecules. Most smells are made up not of a single molecule but of multiple molecules. When you come home and recognize the smell of your family’s best pulled pork, your brain isn’t recognizing the pulled-pork molecule (there is no such thing). Rather, it is recognizing a particular soup of many molecules that activates a symphony of olfactory neurons. Any given smell is represented by a pattern of activated olfactory neurons. In summary, smell recognition is nothing more than pattern recognition.

Our nematode-like ancestor’s ability to recognize the world was constrained to only the sensory machinery of individual neurons. It could recognize the presence of light by the activation of a single photosensitive neuron or the presence of touch from the activation of a single mechanosensory neuron. Although useful for steering, this rendered a painfully opaque picture of the outside world. Indeed, the brilliance of steering was that it enabled the first bilaterians to find food and avoid predators without perceiving much of anything about the world.

However, most of the information about the world around you can’t be found in a single activated neuron but only in the pattern of activated neurons. You can distinguish a car from a house based on the pattern of photons hitting your retina. You can distinguish the ramblings of a person from the roar of a panther based on the pattern of sound waves hitting your inner ear. And, yes, you can distinguish the smell of a rose from the smell of chicken based on the pattern of olfactory neurons activated in your nose. For hundreds of millions of years, animals were deprived of this skill, stuck in a perceptual prison.

When you recognize that a plate is too hot or a needle too sharp, you are recognizing attributes of the world the way early bilaterians did, with the activations of individual neurons. However, when you recognize a smell, a face, or a sound, you are recognizing things in the world in a way that was beyond early bilaterians; you are using a skill that emerged later in early vertebrates.

are similar but not the same.

How Computers Recognize Patterns

You can unlock your iPhone with your face. Doing this requires your phone to solve the generalization and discrimination problems. Your iPhone needs to be able to tell the difference between your face and other people’s faces, despite the fact that faces have overlapping features (discrimination). And your iPhone needs to identify your face despite changes in shading, angle, facial hair, and more (generalization). Clearly, modern AI systems successfully navigate these two challenges of pattern recognition. How?

The standard approach is the following: Create a network of neurons like in

Figure 7.4:

The hard part is teaching the network how to learn the right weights. The state-of-the-art mechanism for doing this was popularized by Geoffrey Hinton, David Rumelhart, and Ronald Williams in the 1980s. Their method is as follows: If you were training a neural network to categorize smell patterns into egg smells or flower smells, you would show it a bunch of smell patterns and simultaneously tell the network whether each pattern is from an egg or a flower (as measured by the activation of a specific neuron at the end of the network). In other words, you tell the network the correct answer. You then compare the actual output with the desired output and nudge the weights across the entire network in the direction that makes the actual output closer to the desired output. If you do this many times (like, millions of times), the network eventually learns to accurately recognize patterns—it can identify smells of eggs and flowers. They called this learning mechanism backpropagation: they propagate the error at the end back throughout the entire network, calculate the exact error contribution of each synapse, and nudge that synapse accordingly.

The above type of learning, in which a network is trained by providing examples alongside the correct answer, is called supervised learning (a human has supervised the learning process by providing the network with the correct answers). Many supervised learning methods are more complex than this, but the principle is the same: the correct answers are provided, and networks are tweaked using backpropagation to update weights until the categorization of input patterns is sufficiently accurate. This design has proven to work so generally that it is now applied to image recognition, natural language processing, speech recognition, and self-driving cars.

But even one of the inventors of backpropagation, Geoffrey Hinton, realized that his creation, although effective, was a poor model of how the brain actually works. First, the brain does not do supervised learning—you are not given labeled data when you learn that one smell is an egg and another is a strawberry. Even before children learn the words egg and strawberry, they can clearly recognize that they are different. Second, backpropagation is biologically implausible. Backpropagation works by magically nudging millions of synapses simultaneously and in exactly the right amount to move the output of the network in the right direction. There is no conceivable way the brain could do this. So then how does the brain recognize patterns?

The Cortex

Olfactory neurons of fish send their output to a structure at the top of the brain called the cortex. The cortex of simpler vertebrates, like the lamprey fish and reptiles, is made up of a thin sheet of

three layers of neurons.

In the first cortex evolved a new morphology of neuron, the pyramidal neuron, named after their pyramid-like shape. These pyramidal neurons have hundreds of dendrites and receive inputs across thousands of synapses. These were the first neurons designed for the purpose of recognizing patterns.

Olfactory neurons send their signals to the pyramidal neurons of the cortex. This network of olfactory input to the cortex has two interesting properties. First, there is a large dimensionality expansion—a small number of olfactory neurons connect to a much larger number of cortical neurons. Second, they connected sparsely; a given olfactory cell will connect to only a subset of these cortical cells. These two seemingly innocuous features of wiring may solve the discrimination problem.

Using

networks to do math. Not complicated math, just addition. They were neuroscientists at Johns Hopkins, and they were both interested in how neural networks stored and maintained memories. This was before artificial neural networks had entered the mainstream, before they had proven their many practical uses—neural networks were still something to be probed for missing capabilities and unseen limitations.

Cohen and McCloskey converted numbers into patterns of neurons, then trained a neural network to do addition by transforming two input numbers (e.g., 1 and 3) into the correct output number (in this case, 4). They first taught the network to add ones (1+2, 1+3, 1+4, and so on) until it got good at it. Then they taught the same network to add twos (2+1, 2+2, 2+3, and so on) until it got good at this as well.

But then they noticed a problem. After they taught the network to add twos, it forgot how to add ones. When they propagated errors back through the network and updated the weights to teach it to add twos, the network had simply overridden the memories of how to add ones. It successfully learned the new task at the expense of the previous task.

Cohen and McCloskey referred to this property of artificial neural networks as the problem of catastrophic forgetting. This was not an esoteric finding but a ubiquitous and devastating limitation of neural networks: when you train a neural network to recognize a new pattern or perform a new task, you risk interfering with the network’s previously learned patterns.

How do modern AI systems overcome this problem? Well, they don’t yet. Programmers merely avoid the problem by freezing their AI systems after they are trained. We don’t let AI systems learn things sequentially; they learn things all at once and then stop learning.

The artificial neural networks that recognize faces, drive cars, or detect cancer in radiology images do not learn continually from new experiences. As of this book going to print, even ChatGPT, the famous chatbot released by OpenAI, does not continually learn from the millions of people who speak to it. It too stopped learning the moment it was released into the world. These systems are not allowed to learn new things because of the risk that they will forget old things (or learn the wrong things). So modern AI systems are frozen in time, their parameters locked down; they are allowed to be updated only when retrained from scratch with humans meticulously monitoring their performance on all the relevant tasks.

The humanlike artificial intelligences we strive to create are, of course, not like this. Rosey from The Jetsons learned as you spoke to her—you could show her how to play a game and she could then play it without forgetting how to play other games.

While we are only just beginning to explore how to make continual learning work, animal brains have been doing so for a long time.

We saw in

the year before.

There are several theories about how vertebrate brains do this. One theory is that the cortex’s ability to perform pattern separation shields it from the problem of catastrophic forgetting; by separating incoming patterns in the cortex, patterns are inherently unlikely to interfere with each other.

Another theory is that learning in the cortex selectively occurs only during moments of surprise; only when the cortex sees a pattern that passes some threshold of novelty are the weights of synapses allowed to change. This enables learned patterns to remain stable for long periods of time, as learning occurs only selectively. There is some evidence that the wiring between the cortex and the thalamus—both structures that emerged alongside each other in early vertebrates—are always measuring the level of novelty between incoming sensory data through the thalamus and the patterns represented in the cortex. If there is a match, then no learning is allowed, hence noisy inputs don’t interfere with existing learned patterns. However, if there is a mismatch—if an incoming pattern is sufficiently new—then this triggers a process of neuromodulator release, which triggers changes in synaptic connections in the cortex, enabling it to now

learn this new pattern.

We do not yet understand exactly how simple vertebrate brains, like those of fish, reptiles, and amphibians, are capable of overcoming the challenges of catastrophic forgetting. But the next time you spot a fish, you will be in the presence of the answer, hidden in its small cartilaginous head.

The Invariance Problem

Look at the two objects below.

As you view each object, a specific pattern of neurons in the backs of your eyes light up. The minuscule half-millimeter-thick membrane in the back of the eye—the retina—contains over one hundred million neurons of five different types. Each region of the retina receives input from a different location of the visual field, and each type of neuron is sensitive to different colors and contrasts. As you view each object, a unique pattern of neurons activates a symphony of spikes. Like the olfactory neurons that make up a smell pattern, the neurons in the retina make up a visual pattern; your ability to see exists only in your ability to recognize these visual patterns.

The activated neurons in the retina send their signals to the thalamus, which then sends these signals to the part of the cortex that processes visual input (the visual cortex). The visual cortex decodes and memorizes the visual pattern the same way the olfactory cortex decodes and memorizes smell patterns. This is, however, where the similarity between sight and smell ends.

Look at the objects below. Can you identify which shapes are the same as the ones in the first picture?

The fact that it is so effortlessly obvious to you that the objects in

presented the cats with different visual stimuli. They presented dots, lines, and various shapes in different locations in the cats’ visual field. They wanted to know how the cortex encoded visual input.

In mammal brains (cats, rats, monkeys, humans, et cetera), the part of the cortex that first receives input from the eye is called V1 (the first visual area). Hubel and Wiesel discovered that individual neurons in V1 were surprisingly selective with what they respond to. Some neurons were activated only by vertical lines at a specific location in a cat’s visual field. Other neurons were activated only by horizonal lines at some other location, and still others by 45-degree lines at a different location. The entire surface area of V1 makes up a map of the cat’s full field of view, with individual neurons selective for lines of specific orientations at each location.

V1 decomposes the complex patterns of visual input into simpler features, like lines and edges. From here, the visual system creates a hierarchy: V1 sends its output to a nearby region of cortex called V2, which then sends information to an area called V4, which then sends information to an area called IT.

Neurons at progressively higher levels of this cortical hierarchy become sensitive to progressively more sophisticated features of visual stimuli—neurons in V1 are primarily activated by basic edges and lines, neurons in V2 and V4 are sensitive to more complex shapes and objects, and neurons in IT are sensitive to complex whole objects such as specific faces. A neuron in V1 responds only to input in a specific region of one’s visual field; in contrast, a neuron in IT can detect objects across any region of the eye. While V1 decomposes pictures into simple features, as visual information flows up the hierarchy, it is pieced back together into whole objects.

In the late 1970s, well over twenty years after Hubel and Wiesel’s initial work, a computer scientist by the name of Kunihiko Fukushima was trying to get computers to recognize objects in pictures. Despite his best attempts, he couldn’t get standard neural networks, like those depicted earlier in the chapter, to successfully do it; even small changes in the location, rotation, or size of an object activated entirely different sets of neurons, which blinded networks to generalizing different patterns to the same object—a square over here would be incorrectly perceived as different from the same square over there. He had stumbled on the invariance problem. And he knew that somehow, brains solved it.

Fukushima had spent the prior four years working in a research group that included several neurophysiologists, and so he was familiar with the work of Hubel and Wiesel. Hubel and Wiesel had discovered two things. First, visual processing in mammals was hierarchical, with lower levels having smaller receptive fields and recognizing simpler features, and higher levels having larger receptive fields and recognizing more complex objects. Second, at a given level of the hierarchy, neurons were all sensitive to similar features, just in different places. For example, one area of V1 would look for lines at one location, and another area would look for lines for another location, but they were all looking for lines.

Fukushima had a hunch that these two findings were clues as to how brains solved the invariance problem, and so Fukushima invented a new architecture of artificial neural networks, one designed to capture these two ideas

discovered by Hubel and Wiesel. His architecture departed from the standard approach of taking a picture and throwing it into a fully connected neural network. His architecture first decomposed input pictures into multiple feature maps, like V1 seemed to do. Each feature map was a grid that signaled the location of a feature—such as vertical or horizontal lines—within the input picture. This process is called a convolution, hence the name applied to the type of network that Fukushima had invented: convolutional neural networks.

Figure 7.10: A convolutional neural network

Figure designed by Max Bennett. The dog photo is from Oscar Sutton (purchased on Unsplash).

tap pictures to get food. She presented the goldfish with two pictures. Whenever the fish tapped specifically a picture of a frog, she gave the fish food. Fish quickly learned to swim right up to the frog picture whenever it was presented. DeLong then changed the experiment. She presented the picture of the same frog but from new angles that the fish had never seen before. If fish were unable to recognize the same object from different angles, they would have treated this like any other photo. And yet, amazingly, the fish swam right up to the new frog picture, clearly able to immediately recognize the frog despite the new angle, just as you recognized the 3D objects a few pages ago.

How the fish brain does this is not understood. While auto-association captures some principles of how pattern recognition works in the cortex, clearly even the cortex of fish is doing something far more sophisticated. Some theorize that the vertebrate brain’s ability to solve the invariance problem derives not from the unique cortical structures in mammals, but from the complex interactions between the cortex and the thalamus, interactions that have been present since the first vertebrates. Perhaps the thalamus—a ball-shaped structure at the center of the brain—operates like a three-dimensional blackboard, with the cortex providing initial sensory input and the thalamus intelligently routing this sensory information around other areas of the cortex, together rendering full 3D objects from 2D input, thereby flexibly able to recognize

rotated and translated objects.

Perhaps the best lesson from CNNs is not the success of the specific assumptions they attempt to emulate—such as translational invariance—but the success of assumptions themselves. Indeed, while CNNs may not capture exactly how the brain works, they reveal the power of a good inductive bias. In pattern recognition, it is good assumptions that make learning fast and efficient. The vertebrate cortex surely has such an inductive bias, we just don’t know what it is.

In some ways, the tiny fish brain surpasses some of our best computer-vision systems. CNNs require incredible amounts of data to understand changes in rotations and 3D objects, but a fish seems to recognize new angles of a 3D object in one shot.

In the predatory arms race of the Cambrian, evolution shifted from arming animals with new sensory neurons for detecting specific things to arming animals with general mechanisms for recognizing anything.

With this new ability of pattern recognition, vertebrate sensory organs exploded with complexity, quickly flowering into their modern form. Noses evolved to detect chemicals; inner ears evolved to detect frequencies of sound; eyes evolved to detect sights. The coevolution of the familiar sensory organs and the familiar brain of vertebrates is not a coincidence—they each facilitated the other’s growth and complexity. Each incremental improvement to the brain’s pattern recognition expanded the benefits to be gained by having more detailed sensory organs; and each incremental improvement in the detail of sensory organs expanded the benefits to be gained by more sophisticated pattern recognition.

In the brain, the result was the vertebrate cortex, which somehow recognizes patterns without supervision, somehow accurately discriminates overlapping patterns and generalizes patterns to new experiences, somehow continually learns patterns without suffering from catastrophic forgetting, and somehow recognizes patterns despite large variances in its input.

The elaboration of pattern recognition and sensory organs, in turn, also found themselves in a feedback loop with reinforcement learning itself. It is also not a coincidence that pattern recognition and reinforcement learning evolved simultaneously in evolution. The greater the brain’s ability to learn arbitrary actions in response to things in the world, the greater the benefit to be gained from recognizing more things in the world. The more unique objects and places a brain can recognize, the more unique actions it can learn to take. And so the cortex, basal ganglia, and sensory organs evolved together, all emerging from the same machinations of reinforcement learning.

Montezuma’s Revenge.

In Montezuma’s Revenge, you start in a room filled with obstacles. In each direction is another room, each with its own obstacles. There is no sign or clue as to which direction is the right way to go. The first reward is earned when you find your way to a hidden door in a faraway hidden room. This makes the game particularly hard for reinforcement learning systems: the first reward occurs so late in the game that there is no early nudging of what behavior should be reinforced or punished. And yet somehow, of course, humans beat this game.

It wasn’t until 2018 when an algorithm was developed that finally completed level one of Montezuma’s Revenge. This new algorithm, developed by Google’s DeepMind, accomplished this feat by adding something familiar that was missing from Sutton’s original TD learning algorithm: curiosity.

Sutton had always known that a problem with any reinforcement learning system is something called the exploitation-exploration dilemma. For trial-and-error learning to work, agents need to, well, have lots of trials from which to learn. This means that reinforcement learning can’t work by just exploiting behaviors they predict lead to rewards; it must also explore new behaviors.

In other words, reinforcement learning requires two opponent processes—one for behaviors that were previously reinforced (exploitation) and the other for behaviors that are new (exploration). These choices are, by definition, opposing each other. Exploitation will always drive behavior toward known rewards, and exploration will always drive toward what is unknown.

In early TD learning algorithms, this trade-off was implemented in a crude way: these AI systems spontaneously—say, 5 percent of the time—did something totally random. This worked okay if you were playing a constrained game with only so many next moves, but it worked terribly in a game like Montezuma’s Revenge, where there were practically an infinite number of directions and places you could go.

There is an alternative approach to tackling the exploitation-exploration dilemma, one that is both beautifully simple and refreshingly familiar. The approach is to make AI systems explicitly curious, to reward them for exploring new places and doing new things, to make surprise itself reinforcing. The greater the novelty, the larger the compulsion to explore it. When AI systems playing Montezuma’s Revenge were given this intrinsic motivation to explore new things, they behaved very differently—indeed, more like a human player. They became motivated to explore areas, go to new rooms, and expand throughout the map. But instead of exploring through random actions, they explored deliberately; they specifically wanted to go to new places and to do new things.

Even though there are no explicit rewards until you get past all the rooms in level one, these AI systems didn’t need any external rewards to explore. They were motivated on their own. Simply finding their way to a new room was valuable in and of itself. Armed with curiosity, suddenly these models started making progress, and they eventually beat level one.

The importance of curiosity in reinforcement learning algorithms suggests that a brain designed to learn through reinforcement, such as the brain of early vertebrates, should also exhibit curiosity. And indeed, evidence suggests that it was early vertebrates who first became curious. Curiosity is seen across all vertebrates, from fish to mice to monkeys

to human infants. In vertebrates, surprise itself triggers the release of dopamine, even if there is

no “real” reward. And yet, most invertebrates do not exhibit curiosity; only the most advanced invertebrates, such as insects and

cephalopods, show curiosity, a trick that evolved independently and

wasn’t present in early bilaterians.

The emergence and mechanisms of curiosity help explain gambling, which is an irrational oddity of vertebrate behavior. Gamblers violate Thorndike’s law of effect—they continue to gamble their money away despite the fact that the expected reward is negative.

B. F. Skinner was the first to realize that rats will gamble. The best way to get a rat to obsessively push a lever for food is not to have the lever release food pellets every time it is pressed; instead, it is to have the lever randomly release food pellets. Such variable-ratio reinforcement makes rats go crazy; they endlessly push the lever, seemingly obsessed with seeing if just one more push might be the one to produce a pellet. Even when the total number of pellets released are identical overall, such variable-ratio reinforcement leads to far more lever pushes than fixed-ratio reinforcement.

Fish exhibit this effect as well.

One explanation for this is that vertebrates get an extra boost of reinforcement when something is surprising. To make animals curious, we evolved to find surprising and novel things reinforcing, which drives us to pursue and explore them. This means that even if the reward of an activity is negative, if it is novel, we might pursue it anyway.

Games of gambling are carefully designed to exploit this. In games of gambling, you don’t have a 0 percent chance of winning (which would lead you not to play); you have a 48 percent chance of winning, high enough to make it possible, uncertain enough to make it surprising when you win (giving you a dopamine boost), and low enough so that the casino will, in the long run, suck you dry.

Our Facebook and Instagram feeds exploit this as well. With each scroll, there is a new post, and randomly, after some number of scrolls, something interesting shows up. Even though you might not want to use Instagram, the same way gamblers don’t want to gamble or drug addicts don’t want to use anymore, the behavior is subconsciously reinforced, making it harder and harder to stop.

Gambling and social feeds work by hacking into our five-hundred-million-year-old preference for surprise, producing a maladaptive edge case that evolution has not had time to account for.

Curiosity and reinforcement learning coevolved because curiosity is a requirement for reinforcement learning to work. With the newfound ability to recognize patterns, remember places, and flexibly change behavior based on past rewards and punishments, the first vertebrates were presented with a new opportunity: for the first time, learning became, in and of itself, an extremely valuable activity. The more patterns a vertebrate recognized and the more places she remembered, the better she would survive. And the more new things she tried, the more likely she was to learn the correct contingencies between her actions and their corresponding outcomes. And so it was 500 million years ago in the tiny brain of our fish-like ancestors when curiosity first emerged.

directly to the container with the food.

Fish are not learning some fixed rule of Always turn left when I see this object—they navigate to the correct location no matter where in the tank they are initially placed. And they are not learning some fixed rule of Swim toward this image or smell of food; fish will go back to the correct container even if you don’t put any food back in the container. In other words, even if every container is exactly identical because none of them have any food at all, fish still correctly identify which container is currently placed at the exact location that previously contained food.

The only clue as to which container previously held the food was the walls of the tank itself, which had markings to designate specific sides. Thus, fish somehow identified the correct container based solely on the container’s location relative to the landmarks on the side of the tank. The only way fish could have accomplished this is by building a spatial map—an internal model of the world—in their minds.

The ability to learn a spatial map is seen across vertebrates. Fish, reptiles, mice, monkeys, and humans all do this. And yet simple bilaterians like nematodes are incapable of learning such a spatial map—they cannot remember the location of

one thing relative to another thing.

Even many advanced invertebrates such as bees and ants are unable to solve spatial tasks. Consider the following ant study. Ants form a path from a nest to food and another path from the food source to their nest. They return to the nest with scavenged food, then leave again empty-handed to acquire more food. Suppose you took one of the ants on its way back to the nest and placed it on the path that is leaving the nest. This ant clearly wants to go back to the nest, but is now placed in a location from which it has only ever navigated away from the nest. If the ant had an internal model of space, it would realize that the fastest way home is to just turn around and go in the opposite direction. That is what a fish would do. But if instead the ant learned only a series of movements (turn right at cue X, turn left at cue Y), then it would just dutifully begin the loop all over again. Indeed, ants go

through the entire loop again. Ants navigate by following a set of rules of when to turn where, not by constructing a map of space.

Your Inner Compass

Here’s another test you can do on yourself: Sit in one of those swivel chairs, close your eyes, ask someone to turn the chair, and then guess what direction of the room you are facing before opening your eyes. You will be amazingly accurate. How did your brain do this?

Deep in your inner ear are semicircular canals, small tubes filled with fluid. These canals are lined with sensory neurons that float in this fluid and activate whenever they detect movement. The semicircular canals are organized in three loops, one for facing forward, one for facing sideways, and one for facing upward. The fluid in each of these canals moves only when you move in that specific dimension. Thus, the ensemble of activated sensory cells signal the direction of head movement. This creates a unique sense—the vestibular sense. This is why you get dizzy if you are spun in a chair continuously—eventually, this overactivates these sensory cells and when you stop turning, they are still active, incorrectly signaling rotation even when you aren’t rotating.

The evolutionary origin of semicircular canals was in early vertebrates, emerging at the same time as reinforcement learning and the ability to construct spatial maps. Modern fish have the same structure in their inner ears, and it enables them to identify when and by how much they are moving.

The vestibular sense is a necessary feature of building a spatial map. An animal needs to be able to tell the difference between something swimming toward it and it swimming toward something. In each case, the visual cues are the same (both show an object coming closer), but each means very different things in terms of movement through space. The vestibular system helps the fish tell the difference: If it starts swimming toward an object, the vestibular system will detect this acceleration. In contrast, if an object starts moving toward it, no such activation will occur.

In the hindbrain of vertebrates, in species as diverse as fish and rats, are what are called “head-direction neurons” that fire only when an animal is

facing a certain direction. These cells integrate visual and vestibular input to create a neural compass. The vertebrate brain evolved, from its very beginnings, to model and navigate three dimensional space.

But if the hindbrain of fish constructs a compass of an animal’s own direction, where is the model of external space constructed? Where does the vertebrate brain store information about the locations of things relative to other things?

The Medial Cortex (aka the Hippocampus)

The

cortex of early vertebrates had three subareas: lateral cortex,

ventral cortex, and medial cortex. The lateral cortex is the area where early vertebrates recognized smells and that would later evolve into the olfactory cortex in early mammals. The ventral cortex is the area where early vertebrates learned patterns of sights and sounds and that would later evolve into the amygdala in early mammals. But folded into the middle of the brain was the third area, the medial cortex.

The medial cortex is the part of cortex that later became the hippocampus in mammals. If you record neurons in the hippocampus of fish as they navigate around, you will find some neurons that activate only when the fish are at a specific location in space, others only when the fish are at a border of a tank, and others only when the fish

are facing specific directions. Visual, vestibular, and head-direction signals propagate to the medial cortex, where they are all mixed together and converted into

a spatial map.

Indeed, if you damage the hippocampus of a fish, it can learn to swim toward or away from cues, but it loses the

ability to remember locations. These fish fail to use distant landmarks to figure out the right direction to

turn in a maze; fail to navigate to specific locations in an open

space to get food; and fail to figure out how to escape a simple room when

given different starting locations.

The function and structure of the hippocampus has been conserved across many lineages of vertebrates. In humans and rats, the hippocampus contains place cells, which are neurons that activate only when an animal is in a specific location in an open maze. Damage to the hippocampus in lizards, rats, and humans

similarly impairs spatial navigation.

Clearly the three-layered cortex of early vertebrates performed computations far beyond simple auto-association. Not only is it also seemingly capable of recognizing objects despite large changes in rotation and scale (solving the invariance problem), but it is also seemingly capable of constructing an internal model of space. To speculate: Perhaps the ability of the cortex to recognize objects despite changes in rotation and its ability to model space are related. Perhaps the cortex is tuned to model 3D things—whether those things are objects or spatial maps.

The evolution of spatial maps in the minds of early vertebrates marked numerous firsts. It was the first time in the billion-year history of life that an organism could recognize where it was. It is not hard to envision the advantage this would have offered. While most invertebrates steered around and executed reflexive motor responses, early vertebrates could remember the places where arthropods tended to hide, how to get back to safety, and the locations of nooks and crannies filled with food.

It was also the first time a brain differentiated the self from the world. To track one’s location in a map of space, an animal needs to be able to tell the difference between “something swimming toward me” and “me swimming toward something.”

And most important, it was the first time that a brain constructed an internal model—a representation of the external world. The initial use of this model was, in all likelihood, pedestrian: it enabled brains to recognize arbitrary locations in space and to compute the correct direction to a given target location from any starting location. But the construction of this internal model laid the foundation for the next breakthrough in brain evolution. What began as a trick for remembering locations would go on to become much more.

to previously inhospitable areas. Plants that resembled today’s trees first developed, growing thick roots and creating stable soils for nearby arthropods to live. In the early Devonian period, plants on land were no more than thirty centimeters tall, but by the end of it, they

were thirty meters tall. It was only at this point that our planet began to appear green from above, as land plants spread across Earth’s surface.

While life for arthropods was horrifying in the sea, it was heavenly on land. Arthropods developed new tricks to meet the needs of life on land, diversifying into what resembled today’s spiders and insects. Unfortunately, as we have seen with today’s problem of climate change, Earth’s biosphere is unforgiving to those who proliferate rapidly and unsustainably. What began as a small oasis for arthropod refugees eventually became an overextended orgy of plant life, triggering a global extinction event that would eradicate close to half of all life.

A Story of Two Great Deaths

History repeats itself.

One and a half billion years ago, the explosion of cyanobacteria suffocated the Earth with carbon dioxide and polluted it with oxygen. Over a billion years later, the explosion of plants on land seems to have committed a similar crime.

The inland march of plants was too rapid for evolution to accommodate and rebalance carbon dioxide levels through the expansion of more CO₂-producing animals. Carbon dioxide levels plummeted, which caused the climate to cool. The oceans froze over and gradually became inhospitable to life. This was

the Late Devonian Extinction, the first great death of this era. There are competing theories of what caused it; some argue that it was not overproliferation of plants but some other natural disaster. In any case, it was from the icy graves of this tragedy that our ancestors emerged from the sea.

Extinction events create opportunities for small niches to transform into dominant strategies. Before the Late Devonian Extinction, our ancestors had found such a niche. Most fish stayed far away from the shore to avoid deadly beaching, a situation where a fish becomes stuck on land as tides recede. Although the risk of beaching made it dangerous to pursue, there was a big nutritional prize to be found close to the shore: the warm earthy puddles were full of small insects and vegetation.

Our ancestors were the first fish to evolve the ability to survive out of water. They developed a pair of lungs that augmented their gills, enabling them to extract oxygen from both water and air. And so our ancestors would use their fins both for swimming in water and for wading themselves short distances on land, traveling from puddle to puddle in search of insects.

When the Late Devonian Extinction Event began to freeze over the oceans, our air-breathing and land-walking ancestors were one of the few warm-water fish to survive. As the food supply in warm waters began to die, our ancestors spent more of their time living in the inland puddles. They lost their gills (and thus their ability to breath underwater), and their webbed fins gave way to fingered hands and feet. They became the first tetrapods (tetra for “four” and pods for “feet”), most closely resembling a modern amphibian such as a salamander.

One evolutionary lineage of tetrapods, who were lucky enough to live in parts of the Earth that still supported these warmer puddles, would maintain this lifestyle for hundreds of millions of years—they would become the amphibians of today. Another lineage abandoned the dying shores and wandered farther inland in search of food. This was the lineage of amniotes—the creatures that developed the ability to lay leathery eggs that could survive out of the water.

The first amniotes probably best resembled a lizard of today. Amniotes found an inland ecosystem abundant with food—insects and plants were everywhere for the feasting. Eventually, the Devonian ice age faded and amniotes spread and diversified to all corners of the Earth. The Carboniferous and Permian eras, which collectively lasted from 350 million years ago to 250 million years ago, saw an explosion of amniotes on land.

Living on land presented unique challenges to the amniotes that their fish cousins never faced. One such challenge was temperature fluctuations. Cycles of the day and season create only muted temperature changes deep in the oceans. In contrast, temperatures can fluctuate dramatically on the surface. Amniotes, like fish, were cold-blooded—their only strategy for regulating their body temperature was to physically relocate to warmer places.

One amniote lineage were the reptiles, who would eventually diversify into dinosaurs, lizards, snakes, and turtles. Most of these reptiles dealt with daily temperature fluctuations by becoming immobile at night. Temperatures were too low for their muscles and metabolisms to function properly, so they simply shut down. The fact that reptiles were shut down for a third of their lives presented an opportunity—creatures that could hunt in the night would reap an incredible feast of motionless lizards.

The other lineage of amniotes were our ancestors: the therapsids. The therapsids differed from reptiles at the time in one important way: they developed warm-bloodedness. Therapsids were the first vertebrates to evolve the ability to use energy to generate their own internal heat.

Figure 10.1: The first therapsid

Original art by Rebecca Gelernter

than you can underwater. Thus, fish opted not to simulate and plan their movements but instead to respond quickly whenever something came at them (hence their large midbrain and hindbrain, and comparatively smaller cortex).

The second speculated requirement is warm-bloodedness. For reasons we will see in the next few chapters, simulating actions is astronomically more computationally expensive and time-consuming than the reinforcement-learning mechanisms in the cortex-basal-ganglia system. The electrical signaling of neurons is highly sensitive to temperature—at lower temperatures, neurons fire much more slowly than at warmer temperatures. This meant that a side effect of warm-bloodedness was that mammal brains could operate much faster than fish or reptile brains. This made it possible to perform substantially more complex computations. This is why reptiles, despite their long-range vision on land, were never endowed with the gift of simulating. The only nonmammals that have shown evidence of the ability to simulate actions

and plan are birds. And birds are, conspicuously, the only nonmammal species alive today that independently evolved warm-bloodedness.

Inside the Brain of the First Mammals

Throughout this several-hundred-million-year-long story, from the emergence of fish onto land to the rise of dinosaurs, there was an expansive diversification of animal shapes, sizes, and organs. And yet, there was one thing that was surprisingly unchanged: brains.

From the early vertebrates to the first tetrapods to reptiles and therapsids, brains were largely stuck in a neural dark age. Evolution settled for, or at least was resigned to, the reinforcement-learning brain of the early vertebrates, and it shifted its focus toward tweaking other biological structures—creating jaws, armor, lungs, more ergonomic bodies, warm-bloodedness, scales, fur, and other such morphological modifications. This is why the brain of a modern fish and that of a modern reptile, despite hundreds of millions of years of evolutionary separation, are remarkably similar.

The neocortex of this early mammal was small and took up only a small fraction of the brain. Most volume was given to the olfactory cortex (early mammals, like many modern mammals, had an incredible sense of smell). But despite the small size of the neocortex in early mammals, it was still the kernel from which human intelligence would arise. In the human brain, the neocortex takes up 70 percent of brain volume. In the breakthroughs that followed, this originally small structure would progressively expand from a clever trick to the epicenter of intelligence.

Figure 11.1: The human neocortex

Original art by Rebecca Gelernter

stimuli elicited what responses.

One of the first observations Mountcastle made was that neurons within a vertical column (about five hundred microns in diameter) of the neocortical sheet seemed to all respond similarly to sensory stimuli, while neurons horizontally farther away did not. For example, an individual column within the visual cortex might contain neurons that all similarly responded to bars of light at specific orientations at a specific location in the visual field. However, neurons within nearby columns responded only to bars of light at different orientations or locations. This same finding has been confirmed within multiple modalities. In rats, there are columns of neocortex that respond only to the touch of a specific single whisker, with each nearby column responding to a completely different whisker. In the auditory neocortex, there are individual columns that are selective for specific frequencies of sound.

The second observation that Mountcastle made was that there were many connections vertically within a column and comparatively fewer connections between columns.

The third and final observation Mountcastle made was that under a microscope, the neocortex looked largely identical everywhere. The auditory neocortex, somatosensory neocortex, and visual neocortex all contain the same types of neurons organized in the same way. And this is true across species of mammals—the neocortex of a rat, a monkey, and a human all look relatively the same under a microscope.

These three facts—vertically aligned activity, vertically aligned connectivity, and observed similarity between all areas of neocortex—led Mountcastle to a remarkable conclusion: the neocortex was made up of a repeating and duplicated microcircuit, what he called the neocortical column. The cortical sheet was just a bunch of neocortical columns packed densely together.

This provided a surprising answer to the question of how one structure can do so many different things. According to Mountcastle, the neocortex does not do different things; each neocortical column does exactly the same thing. The only difference between regions of neocortex is the input they receive and where they send their output; the actual computations of the neocortex itself are identical. The only difference between, for example, the visual cortex and the auditory cortex is that the visual cortex gets input from the retina, and the auditory cortex gets input from the ear.

In the year 2000, decades after Mountcastle first published his theory, three neuroscientists at MIT performed a brilliant

test of Mountcastle’s hypothesis. If the neocortex is the same everywhere, if there is nothing uniquely visual about the visual cortex or auditory about the auditory cortex, then you would expect these areas to be interchangeable. Experimenting on young ferrets, the scientists cut off input from the ears and rewired input from the retina to the auditory cortex instead of the visual cortex. If Mountcastle was wrong, the ferrets would end up blind or visually impaired—input from the eye in the auditory cortex would not be processed correctly. If the neocortex was indeed the same everywhere, then the auditory cortex receiving visual input should work the same way as the visual cortex.

Remarkably, the ferrets could see just fine. And when researchers recorded the area of the neocortex that was typically auditory but was now receiving input from the eyes, they found the area responded to visual stimuli just as the visual cortex would. The auditory and visual cortices are interchangeable.

This was further reinforced by studies of congenitally blind patients whose retinas had never sent any signals to their brains. In these patients, the visual cortex never received input from the eyes. However, if you record the activity of neurons in the visual cortex of congenitally blind humans, you find that the visual cortex has not been rendered a functionally useless region. Instead, it becomes responsive to a multitude of other sensory input, such as sounds and touch. This puts meat on the bone of the idea that people who are blind do, in fact, have superior hearing—the visual cortex becomes repurposed to aid in audition. Again, areas of neocortex seem interchangeable.

Consider stroke patients. When patients have damage to a specific area of neocortex, they immediately lose the function in that area. If the motor cortex is damaged, patients can become paralyzed. If the visual cortex is damaged, patients become partially blind. But over time, function can return. This is usually not the consequence of the damaged area of neocortex recovering; typically, that area of neocortex remains dead forever. Instead, nearby areas of neocortex become repurposed to fulfill the functions of the now-damaged area of neocortex. This too suggests that areas of neocortex are interchangeable.

To those in the AI community, Mountcastle’s hypothesis is a scientific gift like no other. The human neocortex is made up of over ten billion neurons and trillions of connections; it is a hopeless endeavor to try and decode the algorithms and computations performed by such an astronomically massive hairball of neurons. So hopeless that many neuroscientists believe that attempting to decode how the neocortex works is a fruitless endeavor, doomed to fail. But Mountcastle’s theory offers a more hopeful research agenda—instead of trying to understand the entire human neocortex, perhaps we only have to understand the function of the microcircuit that is repeated a million or so times. Instead of understanding the trillions of connections in the entire neocortex, perhaps we only have to understand the million or so connections within the neocortical column. Further, if Mountcastle’s theory is correct, it suggests that the neocortical column implements some algorithm that is so general and universal that it can be applied to extremely diverse functions such as movement, language, and perception across every sensory modality.

The basics of this microcircuit can be seen under a microscope. The neocortex contains six layers of neurons (unlike the three-layered cortex seen in earlier vertebrates). These six layers of neurons are connected in a complicated but beautifully consistent way. There is a specific type of neuron in layer five that always projects to the basal ganglia, the thalamus, and the motor areas. In layer four, there are neurons that always get input directly from the thalamus. In layer six, there are neurons that always project to the thalamus. It is not just a soup of randomly connected neurons; the microcircuit is prewired in a specific way to perform some specific computation.

The question is, of course: What is the computation?

Peculiar Properties of Perception

In the nineteenth century, the scientific study of human perception began in full force. Scientists around the world started probing the mind. How does vision work? How does audition work?

The inquiry into perception began with the use of illusions; by manipulating people’s visual perceptions, scientists uncovered three peculiar properties of perception. And because much of perception, in humans at least, occurs in the neocortex, these properties of perception teach us about how the neocortex works.

Property #1: Filling In

The first thing that became clear to these nineteenth-century scientists was that the human mind automatically and unconsciously fills in missing things. Consider the images in

Property #2: One at a Time

If your mind fills in what it thinks is there based on sensory evidence, what happens if there are multiple ways to fill in what you see? All three of the images in

What is interesting about all these ambiguous pictures is that your brain can see only one interpretation at a time. You cannot see a duck and a rabbit simultaneously, even though the sensory evidence is equally suggestive of both. The mechanisms of perception in the brain, for some reason, require it to pick only one.

This also applies to audition. Consider the “cocktail-party effect.” If you are at a noisy cocktail party, you can tune in to the conversation of the person you are speaking to or the conversation of a nearby group. But you cannot listen to both conversations at the same time. No matter which conversation you tune in to, the auditory input into your ear is identical; the only difference is what your brain infers from that input. You can perceive only a single conversation at a time.

Property #3: Can’t Unsee

What happens when sensory evidence is vague—when it isn’t clear that it can be interpreted as anything meaningful at all? Consider the image in 11.6. If you haven’t seen these before, they will look like nothing—just blobs. If I give you a reasonable interpretation of these blobs, all of a sudden, your perception of them will change.

In the nineteenth century, a German physicist and physician named Hermann von Helmholtz proposed a novel theory to explain these properties of perception. He suggested that a person doesn’t perceive what is experienced; instead, he or she perceives what the brain thinks is there—a process Helmholtz called inference. Put another way: you don’t perceive what you actually see, you perceive a simulated reality that you have inferred from what you see.

This idea explains all three of these peculiar properties of perception. Your brain fills in missing parts of objects because it is trying to decipher the truth that your vision is suggesting (“Is there actually a sphere there?”). You can see only one thing at a time because your brain must pick a single reality to simulate—in reality, the animal can’t be both a rabbit and a duck. And once you see that an image is best explained as a frog, your brain maintains this reality when observing it.

While many psychologists came to agree, in principle,

with Helmholtz’s theory, it would take another century before anyone proposed how Helmholtz’s perception by inference might actually work.

Generative Models: Recognizing by Simulating

In the 1990s, Geoffrey Hinton and some of his students (including the same Peter Dayan that had helped discover that dopamine responses are temporal difference learning signals) set their sights on building an AI system that learned in the way that Helmholtz suggested.

We reviewed in

the Helmholtz machine. The Helmholtz machine was, in principle, similar to other neural networks; it received inputs that flowed from one end to the other. But unlike other neural networks, it also had backward connections that flowed the opposite way—from the end to the beginning.

Hinton tested this network with images of handwritten numbers between 0 and 9. A picture of a handwritten number can be given at the bottom of the network (one neuron for each pixel) and will flow upward and activate a random set of neurons at the top. These activated neurons at the top can then flow back down and activate a set of neurons at the bottom to produce a picture of its own. Learning was designed to get the network to stabilize to a state where input that flows up the network is accurately re-created when it flows back down.

At first, there will be big discrepancies between the values in neurons from the image flowing in and the result flowing out. Hinton designed this network to learn with two separate modes: recognition mode and generative mode. When in recognition mode, information flows up the network (starting from an input picture of a 7 to some neurons at the top), and the backward weights are nudged to make the neurons activated at the top of the network better reproduce the input sensory data (make a good simulated 7). In contrast, when in generative mode, information flows down the network (starting from the goal to produce an imagined picture of a 7), and the forward weights are nudged so that the neurons activated at the bottom of the network are correctly recognized at the top (“I recognize what I just made as a 7”).

Nowhere was this network told the right answer; it was never told what properties make up a 2 or even which pictures were 2s or 7s or any other number. The only data the network had to learn from was pictures of numbers. The question was, of course, would this work? Would this toggling back and forth between recognition and generation enable the network to both recognize handwritten numbers and generate its own unique pictures of handwritten numbers without ever having been told the right answer?

Amazingly, it did; it

learned on its own. When these two processes toggle back and forth, the network magically stabilizes. When you give it a picture of the number 7, it will be able to, for the most part, create a similar image of a 7 on the way down. If instead you give it an image of an 8, it will be able to regenerate an input image of an 8.

This might not seem particularly impressive. You gave a network a picture of a number, and it spit out a picture of that same number—what’s the big deal? There are three attributes of this network that are groundbreaking. First, the top of this network now reliably “recognizes” imperfectly handwritten letters without any supervision. Second, it generalizes impressively well; it can tell that two differently handwritten pictures of 7s are both a 7—they will activate a similar set of neurons at the top of the network. And third, and most important, this network can now generate novel pictures of handwritten numbers. By manipulating neurons at the top of this network, you can create lots of handwritten 7s or handwritten 4s or any number it has learned. This network has learned to recognize by generating its own data.

The Helmholtz machine was an early proof of concept of a much broader class of models called generative models. Most modern generative models are more complicated than the Helmholtz machine, but they share the essential property that they learn to recognize things in the world by generating their own data and comparing the generated data to the actual data.

If you aren’t impressed with the generation of small pixelated handwritten numbers, consider how far these generative models have come since 1995. As this book is going to print, there is an active website called

thispersondoesnotexist.com. Every time you refresh the page, you will see a picture of a different person. The reality is more shocking: every time you reload the page, a generative model creates a completely new, never before seen, made-up face. The faces you see do not exist.

What is so amazing about these generative models is that they learn to capture the essential features of the input they are given without any supervision. The ability to generate realistic novel faces requires the model to understand the essence of what constitutes a face and the many ways it can vary. Just as activating various neurons at the top of the Helmholtz machine can generate images of different handwritten numbers, if you activate various neurons at the top of this generative model of faces, you can control what types of faces it generates. If you change the value of one set of neurons, the network will spit out the same face but rotated. If you change the value of a different set of neurons, it will add a beard or change the age or alter the color of the hair (see

Figure 11.9: StyleGAN2 from thispersondoesnotexist.com

Pictures from thispersondoesnotexist.com

Evidence for this is seen in filling-in, one-at-a-time, can’t-unsee visual illusions; evidence is seen in the wiring of the neocortex itself, which has been shown to have many properties consistent

with a generative model; and evidence is seen in the surprising symmetry—the ironclad inseparability—between perception and imagination that is found in both generative models and the neocortex. Indeed, the neocortex as a generative model explains more than just visual illusions—it also explains why humans succumb to hallucinations, why we dream and sleep, and even the inner workings of imagination itself.

Hallucinating, Dreaming, and Imagining: The Neocortex as a Generative Model

People whose eyes stop sending signals to their neocortex, whether due to optic-nerve damage or retinal damage, often get something called Charles Bonnet syndrome. You would think that when someone’s eyes are disconnected from their brain, they would no longer see. But the opposite happens—for several months after going blind, people start seeing a lot. They begin to hallucinate. This phenomenon is consistent with a generative model: cutting off sensory input to the neocortex makes it unstable. It gets stuck in a drifting generative process in which visual scenes are simulated without being constrained to actual sensory input—thus you hallucinate.

Some neuroscientists refer to perception, even when it is functioning properly, as a

“constrained hallucination.” Without sensory input, this hallucination becomes unconstrained. In our example of the Helmholtz machine, this is like randomly activating neurons at the top of the network and producing hallucinated images of numbers without ever grounding these hallucinations in real sensory input.

This idea of perception as a constrained hallucination is, of course, exactly what Helmholtz meant by inference and exactly what a generative model is doing. We match our inner hallucination of reality to the sensory data we are seeing. When the visual data suggests there is a triangle in a picture (even if there is not actually a triangle there), we hallucinate a triangle, hence the filling-in effect.

Generative models may also explain why we dream and why we need sleep. Most animals sleep, and it has numerous benefits, such as saving energy; but only mammals and birds show unequivocal evidence of dreaming as measured by the

presence of REM sleep. And it is only mammals and birds who exhibit hallucinations and disordered perception if deprived of sleep. Indeed, birds seem to have independently evolved their own neocortex-like structure.

The neocortex (and presumably the bird equivalent) is always in an unstable balance between recognition and generation, and during our waking life, humans spend an unbalanced amount of time recognizing and comparatively less time generating. Perhaps dreams are a counterbalance to this, a way to stabilize the generative model through a

process of forced generation. If we are deprived of sleep, this imbalance of too much recognition and not enough generation eventually becomes so severe that the generative model in the neocortex becomes unstable. Hence, mammals start hallucinating, recognition becomes distorted, and the difference between generation and recognition gets blurred. Fittingly, Hinton even called the learning algorithm to train his Helmholtz machine a “wake-sleep algorithm.” Recognition step was when the model was “awake”; the generation step was when the model was “asleep.”

Many features of imagination in mammals are consistent with what we would expect from a generative model. It is easy, even natural, for humans to imagine things that they are not currently experiencing. You can imagine the dinner you ate last night or imagine what you will be doing later today. What are you doing when you are imagining something? This is just your neocortex in generation mode. You are invoking a simulated reality in your neocortex.

The most obvious feature of imagination is that you cannot imagine things and recognize things simultaneously. You cannot read a book and imagine yourself having breakfast at the same time—the process of imagining is inherently at odds with the process of experiencing actual sensory data. In fact, you can tell when someone is imagining something by looking at that person’s pupils—when people are imagining things, their pupils dilate as their brains stop

processing actual visual data. People become pseudo-blind. As in a generative model, generation and recognition cannot be performed simultaneously.

Further, if you record neocortical neurons that become active during recognition (say, neurons that respond to faces or houses), those exact same neurons become active when you simply

imagine the same thing. When you imagine moving certain body parts, the same area activates as if you were actually

moving the body parts. When you imagine certain shapes, the same areas of visual cortex become activated as when you see those shapes. In fact, this is so consistent that neuroscientists can decode what people are imagining simply by recording their neocortical activity (and as evidence that dreaming and imagination are the same general process, scientists can also accurately decode people’s dreams by

recording their brains). People with neocortical damage that impairs certain sensor data (such as being unable to recognize objects on the left side of the visual field) become equally impaired at simply imagining features of that same sensory data (they struggle even to imagine things in

the left visual field).

None of this is an obvious result. Imagination could have been performed by a system separate from recognition. But in the neocortex, this is not the case—they are performed in the exact same area. This is exactly what we would expect from a generative model: perception and imagination are not separate systems but two sides of the same coin.

Predicting Everything

One way to think about the generative model in the neocortex is that it renders a simulation of your environment so that it can predict things before they happen. The neocortex is continuously comparing the actual sensory data with the data predicted by its simulation. This is how you can immediately identify anything surprising that occurs in your surroundings.

As you walk down the street, you are not paying attention to the feelings of your feet. But with every movement you make, your neocortex is passively predicting what sensory outcome it expects. If you placed your left foot down and didn’t feel the ground, you would immediately look to see if you were about to fall down a pothole. Your neocortex is running a simulation of you walking, and if the simulation is consistent with sensor data, you don’t notice it, but if its predictions are wrong, you do.

Brains have been making predictions since early bilaterians, but over evolutionary time, these predictions became more sophisticated. Early bilaterians could learn that the activation of one neuron tended to precede the activation of another neuron and could thereby use the first neuron to predict the second. This was the simplest form of prediction. Early vertebrates could use patterns in the world to predict future rewards. This was a more sophisticated form of prediction. Early mammals, with the neocortex, learned to predict more than just the activation of reflexes or future rewards; they learned to predict everything.

The neocortex seems to be in a continuous state of predicting all its sensory data. If reflex circuits are reflex-prediction machines, and the critic in the basal ganglia is a reward-prediction machine, then the neocortex is a world-prediction machine—designed to reconstruct the entire three-dimensional world around an animal to predict exactly what will happen next as animals and things in their surrounding world move.

Somehow the neocortical microcircuit implements such a general system that it can render a simulation of many types of input. Give it visual input and it will learn to render a simulation of the visual aspects of the world; give it auditory input and it will learn to render a simulation of auditory aspects of the world. This is why the neocortex looks the same everywhere. Different subregions of neocortex simulate different aspects of the external world based on the input they receive. Put all these neocortical columns together, and they make a symphony of simulations that render a rich three-dimensional world filled with objects that can be seen, touched, and heard.

How the neocortex does this is still a mystery. At least one possibility is that it is prewired to make a set of clever assumptions. Modern AI models are often viewed as narrow—that is, they’re able to work in a narrow set of situations they are specifically trained for. The human brain is considered general—it is able to work in a broad set of situations. The research agenda has therefore been to try and make AI more general. However, we might have it backward. One of the reasons why the neocortex is so good at what it does may be that, in some ways, it is far less general than our current artificial neural networks. The neocortex may make explicit narrow assumptions about the world, and it may be exactly these assumptions that enable it to be so general.

The Evolution of Prediction

PREDICTION IN EARLY BILATERIANS	PREDICTION IN EARLY VERTEBRATES	PREDICTION IN EARLY MAMMALS
Predict reflex activation	Predict future rewards	Predict all sensory data
Reflex circuits	Cortex and basal ganglia	Neocortex

For example, the neocortex may be prewired to assume that incoming sensor data, whether visual, auditory, or somatosensory, represent three-dimensional objects that exist separately from ourselves and can move on their own. Therefore, it does not have to learn about space, time, and the difference between the self and others. Instead, it tries to explain all incoming sensory information it receives by assuming it must have been derived from a 3D world

that unfolds over time.

This provides some intution about what Helmholtz meant by inference—the generative model in the neocortex tries to infer the causes of its sensory input. Causes are just the inner simulated 3D world that the neocortex believes best matches the sensory input it is being given. This is also why generative models are said to try to explain their input—your neocortex attempts to render a state of the world that could produce the picture that you are seeing (e.g., if a frog was there, it would “explain” why those shadows look the way they do).

But why do this? What is the point of rendering an inner simulation of the external world? What value did the neocortex offer these ancient mammals?

There are many ongoing debates about what is missing in modern AI systems and what it will take to get AI systems to exhibit human-level intelligence. Some believe the key missing pieces are language and logic. But others, like Yann LeCun, head of AI at Meta, believe they are something else, something more primitive, something that evolved much earlier. In LeCun’s words:

We humans give way too much importance to language and symbols as the substrate of intelligence. Primates, dogs, cats, crows, parrots, octopi, and many other animals don’t have human-like languages, yet exhibit intelligent behavior beyond that of our best AI systems. What they do have is an ability to learn powerful “world models” that allow them to predict the consequences of their actions and to search for and plan actions to achieve a goal. The ability to learn such world models is what’s missing

from AI systems today.

The simulation rendered in the neocortices of mammals (and perhaps in similar structures of birds or even octopuses) is exactly this missing “world model.” The reason the neocortex is so powerful is not only that it can match its inner simulation to sensory evidence (Helmholtz’s perception by inference) but, more important, that its simulation can be independently explored. If you have a rich enough inner model of the external world, you can explore that world in your mind and predict the consequences of actions you have never taken. Yes, your neocortex enables you to open your eyes and recognize the chair in front of you, but it also enables you to close your eyes and still see that chair in your mind’s eye. You can rotate and modify the chair in your mind, change its colors, change its materials. It is when the simulation in your neocortex becomes decoupled from the real external world around you—when it imagines things that are not there—that its power becomes most evident.

This was the gift the neocortex gave to early mammals. It was imagination—the ability to render future possibilities and relive past events—that was the third breakthrough in the evolution of human intelligence. From it emerged many familiar features of intelligence, some of which we have re-created and surpassed in AI systems, others of which are still beyond our grasp. But all of them evolved in the minuscule brains of the first mammals.

In the coming chapters, we will learn how the neocortex enabled early mammals to perform feats like planning, episodic memory, and causal reasoning. We will learn how these tricks were repurposed to enable fine motor skills. We will learn about how the neocortex implements attention, working memory, and self-control. We will see that it is in the neocortex of early mammals where we will find many of the secrets to human-like intelligence, those that are missing from even our smartest AI systems.

rotated, rescaled, or perturbed.

The core evolutionary function of the neocortex might have been the opposite—recognition might have been the means that unlocked the adaptive benefit of simulating. This would suggest that the original evolutionary function of the neocortex was not recognizing the world—an ability the older vertebrate cortex already had—but instead imagining and simulating the world, an ability the older cortex was lacking.

There were three new abilities that neocortical simulating provided early mammals, all three of which were essential for surviving the one-hundred-and-fifty-million-year predatory onslaught of sharp-toothed dinosaurs.

New Ability #1: Vicarious Trial and Error

In the 1930s, the psychologist Edward Tolman, working at UC Berkeley, was putting rats in mazes to see how they learned. Normal psychology-type work at the time—this was the generation that followed Thorndike. The research paradigm of Thorndike’s law of effect, in which animals repeated behavior that had pleasant consequences, was in full force.

Tolman noticed something odd. When rats reached forks in his mazes where the choice was not obvious—where it wasn’t clear whether they should go left or right—rats would pause and look back and forth for a few seconds

before choosing a direction. This made no sense in the standard Thorndikian view that all learning occurred through trial and error—why would the behavior of pausing and toggling one’s head back and forth have been reinforced?

Tolman made a speculation: The rat was “playing out” each option before taking it. Tolman called this “vicarious trial and error.”

Rats expressed this head toggling behavior only when decisions were hard. One way to make decisions hard is for the costs to be close to the benefits. Suppose you put a rat in a tunnel where it passes various doors, each of which leads to food. And suppose that when rats pass these doors, a specific sound is made that signals how long a rat will have to wait for food if it chooses to walk through the door. One sound signals that rats will have to wait only a few seconds; another sound signals that the rat will have to wait up to half a minute. Once rats learn how all this works, they do not perform their head toggling at the doors with a short delay (they just immediately run in to get the food, presumably thinking, Obviously this is worth the wait) or at doors with a long delay (they just immediately walk past the door: No way this is worth the wait, I’ll just check out the next door). But they do perform the head toggling behavior at the doors with a medium delay (Is this worth the wait or should I

).

Another way to make decisions difficult is to change the rules. If suddenly food is no longer where a rat expects it in the maze, the next time the rat is placed in the maze it will substantially increase its head toggling behavior, seeming

to consider alternative paths. Similarly, suppose a rat enters a maze with two types of food, and suppose recently the rat has eaten a lot of one of those types of food (and hence no longer wants that food), then this head toggling behavior emerges (Do I want to turn left to get X instead of

).

Of course, the fact that you can observe a rat pause and turn its head back and forth doesn’t prove that it is, in fact, imagining going down different paths. And due to this lack of evidence, the idea of vicarious trial and error faded out of favor in the decades that followed Tolman’s observation. It was only recently, in the 2000s, that technology reached the point where ensembles of neurons in the brain of a rat could be recorded in real time as rats navigated their environment. This was the first time that neuroscientists could literally watch what was happening in the brains of rats when they paused and toggled their heads back and forth.

It was David Redish and his student Adam Johnson, neuroscientists at the University of Minnesota, who first probed what was happening in the brain of rats during these choices. At the time, it was well known that when a rat navigates a maze, specific place cells in its hippocampus get activated. This was similar to the spatial map of a fish—specific hippocampal neurons encode specific locations. In a fish, these neurons become active only when the fish is physically present at the encoded location—but when Redish and Johnson recorded these neurons in rats, they found something different: when the rat stopped at the decision point and turned its head back and forth, its hippocampus ceased to encode the actual location of the rat and instead went back and forth rapidly playing out the sequence of place codes that made up both possible future paths from the choice point. Redish could literally see the rat imagining future paths.

How groundbreaking this was cannot be overstated—neuroscientists were peering directly into the brain of a rat, and directly observing the rat considering alternative futures. Tolman was right: the head toggling behavior he observed was indeed rats planning their future actions.

In contrast, the first vertebrates did not plan their actions ahead of time. We can see this by examining their cold-blooded descendants—modern fish and reptiles—who show no evidence of learning by vicarious trial and error.

Consider the detour task. Take a fish and put it in a tank with a transparent barrier in the middle of the tank. Put a small hole in one corner of the barrier so that the fish can get from one side to the other. Let the fish explore the tank, find the hole, and spend some time swimming back and forth. After several days, do something new: Put the fish on one side of the tank and put a treat on the opposite side of the transparent barrier. What happens?

The smart thing to do, if you wanted food, would be to immediately swim away from the treat to the corner of the barrier, go through the opening, and then turn back to get the food on the other side. But this is not what the fish does. The fish runs directly into the transparent barrier trying to get the food. After bopping against the wall enough times, it gives up and continues wandering its environment. Eventually, while wandering the environment, the fish happens to pass through the hole again, but even here it shows no understanding that it can now access the food; the fish does not turn toward the food. Instead, the fish just continues onward through the other side of the tank. Only when it happens to turn back and see the food again does it excitedly dart toward it. Indeed, it takes the same amount of time to find the food for a fish who has had many experiences navigating between each side of the barrier, as it does for a fish that has never experienced getting to the other side of

the barrier at all.

Why is this? While the fish had swum through the hole to get to the other side of the tank before, it had never learned that the path through the hole provided dopamine. Trial-and-error learning had never trained the fish’s basal ganglia that when it saw food across the transparent barrier, it should take the action of swimming through the hole to get the food.

This is a critical problem of learning only by doing: Although the fish had learned the path through the hole, it had never taken that path to get food before. So when it saw food, all it could do was generate an “approach” signal directly toward the food. Rats, however, are much savvier. In such detour tasks, they dramatically outperform fish. Both rats and fish will initially run up to the transparent barrier to try and get food. But rats are much better at figuring out how to

navigate around a barrier. And a rat who has explored a map well—who knows how to get to the other side of a transparent barrier (even if doing so was never rewarded)—will get to the other side much faster than will a rat who has never navigated around the barrier before. This reveals one of the benefits of vicarious trial and error: once a rat has a world model of their environment, they can rapidly mentally explore it until they find a way to get around obstacles to get what they want.

Another problem with the older strategy of learning by doing is that sometimes past rewards are not predictive of current rewards because an animal’s internal state has changed. For example, put a rat in a maze where one side provides overly salty food and the other side provides normal food. Let the rat navigate this maze normally and try the overly salty food (which it will hate and then avoid) and the normal food (which the rat will enjoy). Now suppose you put the rat back in that situation but with a twist: You make it severely salt-deprived. What does the rat do?

It immediately

runs toward the salt. This is remarkable because the rat is now running toward a part of the maze that was previously negatively reinforced. This is possible only because the rat “simulated” each path and realized—vicariously—that the overly salty food would now be hugely rewarding. In other words, the path toward salt was vicariously reinforced before the rat even acted. I am unaware of any studies showing that fish or reptiles can perform such a task.

New Ability #2: Counterfactual Learning

Humans spend a painful amount of time wallowing in regret. Questions you might hear from an average human conversation: “What would life have been like if I had said yes when Ramirez offered to throw our lives away and move to Chile to work on his farm?” “What if I had followed my dream and pursued baseball instead of this desk job?” “Why did I say that stupid thing at work today? What would have happened had I said something cleverer?”

Buddhists and psychologists alike realize that ruminating about what could have been is a source of great misery for humanity. We cannot change the past, so why torture ourselves with it? The evolutionary roots of this go back to early mammals. In the ancient world, and in much of the world that followed, such ruminating was useful because often the same situation would recur and a better choice could be made.

The type of reinforcement learning we saw in early vertebrates has a flaw: It can only reinforce the specific action actually taken. The problem with this strategy is that the paths that were actually taken are a small subset of all the possible paths that could have been taken. What are the chances that an animal’s first attempt picked the best path?

If a fish swims into a shoal to hunt some invertebrates and comes away with one, and a nearby fellow fish took a different path and came away with four, the first fish won’t learn from that mistake; it will merely reinforce the path taken with the mediocre reward. What fish are missing is the ability to learn from counterfactuals. A counterfactual is what the world would be now if you had made a different choice in the past.

David Redish, having discovered that rats can imagine alternative futures, wanted to see if rats could also imagine alternative past choices. Redish and one of his students Adam Steiner put rats in a circular maze

they called “restaurant row.” Rats ran around counterclockwise and kept passing the same four corridors. At the end of each corridor was a different “restaurant,” containing a different flavor of food (chocolate, cherry, banana). As the rat passed each corridor, a random tone was sounded that signaled the delay before the food would be released if the rat waited at this corridor instead of continuing on to the next. Tone A meant if the rat waited at the current corridor, it would get the food within forty-five seconds; tone B meant it would get the food in five seconds. If a rat chose not to wait, then once it got to the next corridor, it could no longer go back; the food would not be released unless it went all the way around the circle again. This presented rats with a continuous set of irreversible choices. Rats were given an hour to keep trying to get as much of their favorite foods as they could.

Consider the choice presented to the rats at a given corridor: Do I wait here for mediocre banana that the tone just signaled would be released in five seconds, or do I run to the next door, which contains my favorite food, cherry, and gamble that it will also be released quickly? When rats chose to forgo quick access to a banana treat to try the cherry door and the next tone signaled a long wait of forty-five seconds, rats showed all the signs of regretting their choice. They paused and looked back toward the corridor that they had passed and could no longer go back to. And the neurons in the taste area of the neocortex reactivated the representation of banana, showing that rats were literally imagining a world where they had made a different choice and got to eat the banana.

The rats that turned back and reactivated the representation of the forgone choice also ended up changing their future choices. They waited longer the next time and ate other food more hastily to try and get back around the maze to the cherry to try again.

Primates also reason about counterfactuals. Consider an experiment where monkeys were taught to

play rock, paper, scissors. Whenever monkeys lost, their next move was always biased toward the move that would have won in the prior game. If a monkey lost because it chose paper while its opponent chose scissors, next time it is most likely to pick rock (which would have won against scissors). This cannot be explained with the temporal difference learning of early vertebrates. According to the evolutionary framework presented here, if a fish could play rock, paper, scissors, it would not show this effect. If a fish lost after playing paper against scissors, it would become less likely to play paper next time (that action was punished because it led to losing), but the fish would be equally likely to play rock or scissors next time (neither action was punished or reinforced). In contrast, because a monkey can replay what could have been after losing by playing paper, it will realize that if it had played rock it would have won. Thus, monkeys are changing their actions based on counterfactual learning.

The perception of causation may be intricately tied to the notion of counterfactual learning. What we mean when we say “X caused Y” is that in the counterfactual case where X did not occur, then Y

did not occur either. This is how we tease out the difference between correlation and causation. If you saw lightning strike a dry forest, and a fire immediately started, you would say that it was the lightning that caused the fire, not the fire that caused the lightning. You say this because when you imagine the counterfactual case in which lightning did not strike, the fire did not appear. Without counterfactuals, there is no way to distinguish between causation and correlation. You can never know what caused what; you can know only that “X always happens before Y” or “Whenever X happens, Y happens” or “Y has never happened without X happening,” and so on.

Counterfactual learning represented a major advancement in how ancestral brains solved the credit assignment problem. As a reminder, the credit assignment problem is this: When some important event occurs that you want to be able to predict ahead of time, how do you choose what previous actions or events to give credit for having been predictive of the event? In simple terms: A bunch of things happen (a bird chirps, a gust of wind blows, a leaf moves, and lightning strikes), and then a fire appears—what do you give credit for being a good predictor of fire? In early bilaterians, simple tricks like blocking, latent inhibition, and overshadowing drove the logic by which simple predictions and associations were made. In early vertebrates, the evolution of temporal difference learning enabled the basal ganglia to assign credit using changes in future predicted reward; when the critic thinks the situation just got better or worse is when cues or actions are given credit. But in early mammals, with their ability to simulate alternative pasts, credit could also be assigned with the counterfactual. By asking “Would I have lost the game had I not made this move?,” mammals can determine whether a move truly deserves credit for winning the game.

Causation itself may live more in psychology than in physics. There is no experiment that can definitively prove the presence of causality; it is entirely immeasurable. Controlled experiments we run may suggest causation, but they always fall short of proof because you can, in fact, never run a perfectly controlled experiment. Causation, even if real, is always empirically out of reach. In fact, modern experiments in the field of quantum mechanics suggest that causation may not even exist, at least not everywhere. The laws of physics may contain rules of how features of reality progress from one time step to the next without any real causal relationships between things at all. Ultimately, whether causality is real or not, the evolution of our intuitive perception of causality does not derive from its reality but from its usefulness. Causation is constructed by our brains to enable us to learn vicariously from alternative past choices.

The Evolution of Credit Assignment

CREDIT ASSIGNMENT IN EARLY BILATERIANS	CREDIT ASSIGNMENT IN EARLY VERTEBRATES	CREDIT ASSIGNMENT IN EARLY MAMMALS
Credit assigned based on basic rules of blocking, latent inhibition, and overshadowing	Credit assigned based on when the critic predicts changes in future rewards	Credit assigned based on the counterfactual–which previous events or actions, if they had not occurred, would have prevented the subsequent event from occurring (i.e., what truly caused the event?)

While such counterfactual learning and causal reasoning is indeed seen across many mammals, even rats, there is no convincing evidence that fish or reptiles are capable of learning from counterfactuals or reasoning about causation (however, there is evidence of this type

of reasoning in birds). This suggests that, at least in our lineage, this ability first emerged in our mammalian ancestors.

New Ability #3: Episodic Memory

In September 1953, a twenty-seven year old man named Henry Molaison underwent an experimental procedure that removed his entire hippocampus—the source of his debilitating seizures. The surgery was, in one sense, a success: severity of his seizures were markedly reduced and he retained his personality and intellect. But his doctors quickly discovered that the surgery had deprived their patient of something precious: upon waking up, Molaison was entirely unable to produce new memories. He could hold a conversation for a minute or two, but shortly thereafter would forget everything that just happened. Even forty years later, he could fill out crosswords with facts from before 1953, but not with facts that occurred after. Molaison was stuck in 1953.

We don’t review the past only for the purpose of considering alternative past choices; we also review the past to remember previous life events. People can easily recall what they did five minutes ago or what they majored in at college or that funny joke made during a wedding speech. This form of memory, in which we recall specific past episodes of our lives, is called “episodic memory.” This is distinct from, say, procedural memory, where we remember how to do various movements, such as speaking, typing, or throwing a baseball.

But here is the weird thing—we don’t truly remember episodic events. The process of episodic remembering is one of simulating an approximate re-creation of the past. When imagining future events, you are simulating a future reality; when remembering past events, you are simulating a past reality. Both are simulations.

We know this based on two pieces of evidence. First, remembering past events and imagining future events use similar if not the same neural circuitry. The same networks get activated when you’re imagining the future as when

you’re remembering the past. Remembering specific things (faces, houses) reactivates the same neurons in the sensory neocortex as when you actually perceive those things (just as we

saw with imagining things).

The second piece of evidence that episodic memory is just a simulation comes from phenomena associated with such memories. For example, it turns out people’s episodic memories are “filled in” during the remembering process (much like the filling in of visual shapes). This is why episodic memories feel so real but are much less accurate than we think. The clearest demonstrations of episodic memory’s flaws are in eyewitness testimonies. Someone looking at a lineup of potential criminals is often 100 percent sure which member of the lineup committed the crime. However, contrary to our perceptions of how accurate our memories are, it has been shown that eyewitness testimony is terrible: 77 percent of the wrongfully convicted individuals exonerated by the Innocence Project were originally convicted because of

mistaken eyewitness testimony. The distinction between a made-up imagined scene and an actual episodic memory is thin in the neocortex; studies show that repeatedly imagining a past event that did not occur falsely increases a person’s confidence that

the event did occur.

One can test the presence of episodic memory in animals by asking them unexpected questions about recent events. For example, a rat may learn that when presented with a certain maze, it will find food down one path if and only if it encountered food in the prior few minutes; otherwise, it should go down a different path to get food. The fact that this question is asked randomly (by randomly presenting the rat with this maze) makes it hard to see how the TD-learning mechanisms of the first vertebrates would learn this contingency; the maze is equally paired with each direction. And yet rats easily learn, when placed in this maze, to recall whether they recently encountered food and then choose the right corresponding path

to get more food. The only nonmammals that have been demonstrated to have such episodic memory are birds and cephalopods, the two groups of species that seem to have independently evolved their own brain

structures for rendering simulations.

After Molaison’s surgery, he became the most studied neuroscience patient in history: why was the hippocampus required for creating new episodic memories, but not for retrieving old ones? This is an example of evolution repurposing old structures for new purposes. In mammal brains, episodic memory emerges from a partnership between the older hippocampus and the newer neocortex. The hippocampus can quickly learn patterns, but cannot render a simulation of the world; the neocortex can simulate detailed aspects of the world, but cannot learn new patterns quickly. Episodic memories must be stored rapidly, and thus the hippocampus, designed for the rapid pattern recognition of places, was repurposed to also aid in the rapid encoding of episodic memories. Distributed neural activations of sensory neocortex (i.e., simulations) can be “retrieved” by reactivating the corresponding pattern in the hippocampus. Just as rats reactivated place cells in the hippocampus to simulate going down different paths, rats can reactivate these “memory codes” in the hippocampus to rerender simulations of recent events.

MODEL-FREE REINFORCEMENT LEARNING	MODEL-BASED REINFORCEMENT LEARNING
Learns direct associations between a current state and the best actions	Learns a model of how actions affect the world and uses this to simulate different actions before choosing
Faster decisions but less flexible	Slower decisions but more flexible
Emerged in early vertebrates	Emerged in early mammals
Neocortex is not required	Neocortex is required
Example: Habitually going to work by just responding to each cue (traffic light, landmark) as it comes up	Example: Considering different ways to get to work and picking the one that got you there the fastest in your mind.

This dynamic provided a new solution to the catastrophic forgetting problem, whereby neural networks forget old patterns when they learn new ones. By retrieving and replaying recent memories alongside old memories, the hippocampus aids the neocortex in incorporating new memories without disrupting old ones. In AI, this process is called “generative replay” or “experience replay” and has been shown to be an effective solution to catastrophic forgetting. This is why the hippocampus is necessary for creating new memories, but not for retrieving old ones; the neocortex can retrieve memories on its own after a sufficient amount of replay.

All this simulating of futures and pasts has a larger analog in machine learning. The type of reinforcement learning—temporal difference learning—that we saw with

games and many self-driving-car algorithms

are model-free. These systems don’t pause and consider their choices; they immediately act in response to the sensory data they are given.

Model-based reinforcement learning has proven to be more difficult to implement for two reasons.

First, building a model of the world is hard—the world is complex and the information we get about it is noisy and incomplete. This is LeCun’s missing world model that the neocortex somehow renders. Without a world model, it is impossible to simulate actions and predict their consequences.

The second reason model-based reinforcement learning is hard is that choosing what to simulate is hard. In the same paper that Marvin Minsky identified the temporal credit assignment problem as an impediment to artificial intelligence, he also identified what he called the “search problem”: In most real-world situations, it is impossible to search through all possible options. Consider chess. Building a world model of the game of chess is relatively trivial (the rules are deterministic, you know all the pieces, all their moves, and all the squares of the board). But in chess, you cannot search through all the possible future moves; the tree of possibilities in chess has more branching paths than there are atoms in the universe. So the problem is not just constructing an inner model of the external world but also figuring out how to explore it.

And yet clearly, somehow, the brains of early mammals solved the search problem. Let’s see how.

complex games like chess. The problem is that in complex situations, model-free learning—which contains no planning or playing out of possible futures—is not good at finding the moves that don’t look great right now but set you up well for the future.

In 2017, Google’s DeepMind released an AI system called AlphaZero that achieved superhuman performance in not only the game of chess but also the game of Go, beating the world

Go champion Lee Sedol. Go is an ancient Chinese board game that is even more complex than chess; there are a trillion upon trillion times more possible board positions in

Go than in chess.

How did AlphaZero achieve superhuman performance at Go and chess? How did AlphaZero succeed where temporal difference learning could not? The key difference was that AlphaZero simulated future possibilities. Like TD-Gammon, AlphaZero was a reinforcement learning system—its strategies were not programmed into it with expert rules but learned through trial and error. But unlike TD-Gammon, AlphaZero was a model-based reinforcement learning algorithm; AlphaZero searched through possible future moves before deciding what to do next.

After its opponent moved, AlphaZero would pause, select moves to consider, and then play out thousands of simulations of how the entire game might go given those selected moves. After running a set of simulations, AlphaZero might see that it won thirty-five out of the forty imagined games when it made move A, thirty-nine of the forty imagined games when it made move B, and so on for many other possible next moves. AlphaZero could then pick the move where it had won the highest ratio of imagined games.

Doing this, of course, comes with the search problem; even armed with Google’s supercomputers, it would take well over a million years to simulate every possible future move from an arbitrary

board position in Go. And yet AlphaZero ran these simulations within half a second. How? It didn’t simulate the trillions of possible futures; it simulated only a thousand futures. In other words, it prioritized.

There are many algorithms for deciding how to prioritize which branches to search through in a large tree of possibilities. Google Maps uses such an algorithm when it searches for the optimal route from point A to point B. But the search strategy used by AlphaZero was different and offered unique insight into how real brains might work.

We already discussed how in temporal difference learning an actor learns to predict the best next move based on a hunch about the board position, doing so without any planning. AlphaZero simply expanded on this architecture. Instead of picking the single move its actor believed was the best next move, it picked multiple top moves that its actor believed were the best. Instead of just assuming its actor was correct (which it would not always be), AlphaZero used search to verify the actor’s hunches. AlphaZero was effectively saying to the actor, “Okay, if you think move A is the best move, let’s see how the game would play out if we did move A.” And AlphaZero then also explored other hunches of the actor, considering the second and third best moves the actor was suggesting (saying to the actor, “Okay, but if you didn’t take move A, what would your next best hunch be? Maybe move B will turn out even better than you think”).

What is elegant about this is that AlphaZero was, in some sense, just a clever elaboration on Sutton’s temporal difference learning, not a reinvention of it. It used search not to logically consider all future possibilities (something that is impossible in most situations) but to simply verify and expand on the hunches that an actor-critic system was already producing. We will see that this approach, in principle, may have parallels to how mammals navigate the search problem.

While Go is one of the most complex board games, it is still far simpler than the task of simulating futures when moving around in the real world. First, the actions in Go are discrete (from a given board position, there are only about two hundred

possible subsequent next moves), whereas in the real world actions are continuous (there are an infinite number of possible

body and navigational paths). Second, the information about the world in Go is deterministic and complete, whereas in the real world it is noisy and incomplete. And third, the rewards in Go are simple (you either win or lose the game), but in the real world, animals have competing needs that change over time. And so, while AlphaZero was a huge leap forward, AI systems are still far from performing planning in environments with a continuous space of actions, incomplete information about the world, and complex rewards.

However, the most critical advantage of planning in mammalian brains over modern AI systems like AlphaZero is not their ability to plan with continuous action spaces, incomplete information, or complex rewards, but instead simply the mammalian brain’s ability to flexibly change its approach to planning depending on the situation. AlphaZero—which applied only to board games—employed the same search strategy with every move. In the real world, however, different situations call for different strategies. The brilliance of simulation in mammal brains is unlikely to be some special yet-to-be-discovered search algorithm; it is more likely to be the flexibility with which mammal brains employ different strategies. Sometimes we pause to simulate our options, but sometimes we don’t simulate things at all and just act instinctually (somehow brains intelligently decide when to do each). Sometimes we pause to consider possible futures, but other times we pause to simulate some past event or alternative past choices (somehow brains select when to do each). Sometimes we imagine rich details in our plans, playing out each individual detailed subtask, and sometimes we render just the general idea of the plan (somehow brains intelligently select the right granularity of our simulation). How do our brains do this?

Prefrontal Cortex and Controlling the Inner Simulation

In the 1980s, a neuroscientist named Antonio Damasio visited one of his patients—referred to as “L”—who had suffered a stroke. L lay in bed with her eyes open and a blank expression on her face. She was motionless and speechless, but she wasn’t paralyzed. She would, at times, lift up the blanket to cover herself with perfectly fine motor dexterity; she would look over at a moving object, and she could clearly recognize when someone spoke her name. But she did and said nothing. When looking into her eyes, people said it seemed that “she was there but not there.”

Stroke victims with damage to the visual, somatosensory, or auditory neocortex suffer from impairments to perception (such as blindness or deafness). But L showed none of these symptoms; her stroke occurred in a specific region of her prefrontal neocortex. L had developed akinetic mutism—a tragic and bizarre condition caused by damage to certain regions of the prefrontal cortex in which people are able to move and understand things just fine but they don’t move, speak, or care

about anything at all.

After six months, as with many stroke patients, L began to recover as other areas of her neocortex remapped themselves to compensate for the damaged area. As L slowly began to speak again, Damasio asked her about her experience over the prior six months. Although L had little memory of it, she did recall the few days before beginning to speak. She described the experience as not talking because she had nothing to say. She claimed her mind was entirely “empty” and that nothing “mattered.” She claimed that she was fully able to follow the conversations around her, but she “felt

no ‘will’ to reply.” It seems that L had lost all intention.

The neocortex of all mammals can be separated into two halves. The back half is the sensory neocortex, containing visual, auditory, and somatosensory areas. Everything about the neocortex we reviewed in

to be a mammal.

In the first mammals, the entire frontal cortex was just agranular prefrontal cortex. All modern mammals contain an agranular prefrontal cortex inherited from the first mammals. To understand L’s akinetic mutism and how mammals decide when and what to simulate, we must first roll back the evolutionary clock to explore the function of the aPFC in the brains of the first mammals.

It seems that in early mammals, the sensory neocortex was where simulations were rendered, and the frontal neocortex was where simulations were controlled—it is the frontal neocortex that decided when and what to imagine. A rat with a damaged frontal neocortex loses the ability to trigger simulations; it no longer engages in

vicarious trial and error,

episodic-memory recall,

or counterfactual learning. This impairs rats in many ways. They become worse at solving spatial-navigation challenges that require preplanning, such as when they are placed in completely new starting locations in a maze. Rats make lazier choices, often taking the easier paths even if they offer substantially fewer rewards, as if the rats are unable to stop and play out each option and realize that the

effort is worth it. And without episodic memory, they fail to internally recall old memories of dangerous cues and are thereby more likely

to repeat past mistakes.

Even those rats with only partial frontal damage who retain some ability to trigger these simulations still struggle to monitor these simulated “plans” as they unfold. Rats with aPFC damage struggle to remember where they are in an ongoing plan, do actions out of sequence, and unnecessarily

repeat already completed actions. Rats with aPFC damage also become impulsive; they will prematurely respond in tasks that require them to wait and be

patient to get food.

While the frontal and sensory cortices seem to serve different functions (the frontal neocortex triggers simulations, the sensory neocortex renders simulations), they are both different areas of the neocortex and thus should execute the same fundamental computation. This presents a conundrum: How does the frontal neocortex, simply another area of neocortex, do something so seemingly different from the sensory neocortex? Why does a modern human with a damaged aPFC become devoid of intention? How does the aPFC trigger simulations in the sensory neocortex? How does it decide when to simulate something? How does it decide what to simulate?

Predicting Oneself

In a column of the sensory cortex, the primary input comes from external sensors, such as the eyes, ears, and skin. The primary input to the agranular prefrontal cortex, however, comes from the hippocampus, hypothalamus, and amygdala. This suggests that the aPFC treats sequences of places, valence activations, and internal affective states the way the sensory neocortex treats sequences of sensory information. Perhaps, then, the aPFC tries to explain and predict an animal’s own behavior the same way that the sensory neocortex tries to explain and predict the flow of external sensory information?

Perhaps the aPFC is always observing a rat’s basal-ganglia-driven choices and asking, “Why did the basal ganglia choose this?” The aPFC of a given rat might thereby learn, for example, that when the rat wakes up and has these specific hypothalamic activations, it always runs down to the river and consumes water. The aPFC might then learn that the why of such behavior is “to get water.” Then in similar circumstances, the aPFC can predict what the animal will do before the basal ganglia triggers any behavior—it can predict that when thirsty, the animal will run toward nearby water. In other words, the aPFC learns to model the animal itself, inferring the intent of behavior it observes, and uses this intent to predict what the animal will do next.

As philosophically fuzzy as intent might sound, it is conceptually no different from how the sensory cortex constructs explanations of sensory information. When you see a visual illusion that suggests a triangle (even when there is no triangle), your sensory neocortex constructs an explanation of it, which is what you perceive—a triangle. This explanation—the triangle—is not real; it is constructed. It is a computational trick that the sensory neocortex uses to make predictions. The explanation of the triangle enables your sensory cortex to predict what would happen if you reached out to grab it or turned the light on or tried to look at it from another angle.

Frontal vs Sensory Neocortex in the First Mammals

FRONTAL NEOCORTEX	SENSORY NEOCORTEX
A self model	A world model
Gets input from hippocampus, amygala, and hypothalamus	Gets input from sensory organs
“I did this because I want to get to water”	“I see this because there is a triangle right there”
Tries to predict what the animal will do next	Tries to predict what external objects will do next

Recording studies corroborate the idea that the aPFC creates a model of an animal’s goals. If you look at a recording of the aPFC of a rat, you can see patterns of activity that encode the task a rat is performing—with specific populations of neurons selectively firing only at specific locations within a complex task sequence, reliably tracking progress

toward an imagined goal.

What is the evolutionary usefulness of this model of self in the frontal cortex? Why try to “explain” one’s own behavior by constructing “intent”? It turns out, this might be how mammals choose when to simulate things and how to select what to simulate. Explaining one’s own behavior might solve the search problem. Let’s see how.

How Mammals Make Choices

Let’s take the example of a rat navigating a maze and making a choice as to which direction to go when it reaches a fork. Going to the left leads to water, going to the right leads to food. It is these situations when vicarious trial and error occurs, and it occurs in three steps.

Step #1: Triggering Simulation

The columns in the aPFC might always be in one of three states: (1) silent; the behavior it observes is not recognizing any specific intent (like a column in the visual cortex not recognizing anything meaningful in an image); (2) many or all columns in the frontal cortex recognize an intent and predict the same next behavior (“Oh! We are obviously about to go to the left here”); or (3) many columns in the frontal cortex recognize an intent but predict different and inconsistent behaviors (some columns predict “I’m going to go down to the left to get water!” and other columns predict “I’m going to go to the right to get food!”). It might be in this last state, in which aPFC columns don’t agree in their predictions, where the aPFC might be most useful. Indeed, the aPFC of mammals gets most excited when something goes wrong or something unexpected happens

in an ongoing task.

The degree of disagreement of predictions is a measure of uncertainty. This is, in principle, how many state-of-the-art machine learning models measure uncertainty: An ensemble of different models makes predictions, and the more divergent such predictions, the more uncertainty there

is reported to be.

It might be this uncertainty that triggers simulations. The aPFC can trigger a global pause signal through connecting directly to specific parts

of the basal ganglia, and aPFC activation has been shown to correlate

with levels of uncertainty. And as we saw in the last chapter, it is exactly when things change or are hard (i.e., are uncertain) that animals pause to engage in vicarious trial and error. It is also possible that uncertainty could be measured in the basal ganglia; perhaps there are parallel actor-critic systems, each making independent predictions about the next best action, and the divergence in their predictions is what triggers pausing.

Either way, this provides a speculation for how mammal brains tackle the challenge of deciding when to go through the effort of simulating things. If events are unfolding as one would expect, there is no reason to waste time and energy simulating options, and it is easier just to let the basal ganglia drive decisions (model-free learning), but when uncertainty emerges (something new appears, some contingency is broken, or costs are close to the benefits), then simulation is triggered.

Step #2: Simulating Options

Okay, so the rat paused and decided to use simulation to resolve its uncertainty—now what? This brings us back to the search problem. A rat in a maze could do any one of a billion things, so how does it decide what to simulate?

We saw how AlphaZero solved this problem: It played out the top moves it was already predicting were the best. This idea aligns quite nicely with what is known about neocortical columns and the basal ganglia. The aPFC doesn’t sit there combing through every possible action, instead it specifically explores the paths that it is already predicting an animal will take. In other words, the aPFC searches through the specific options that different columns of the aPFC are already predicting. One set of columns predicted going left all the way to water, and another predicted going right, so there are only two different simulations to run.

After an animal pauses, different columns in the aPFC take turns playing out their predictions of what they think the animal will do. One group of columns plays out going left and following that path all the way to water. Another group of columns plays out going right and following that path all the way to get food.

The connectivity between the aPFC and the sensory neocortex is revealing; the aPFC projects extensively to diffuse regions of the sensory cortex and has been shown to dramatically modulate the

activity of the sensory neocortex. And specifically when rats engage in this vicarious trial and error behavior, the activity in the aPFC and the sensory

cortex become uniquely synchronized. One speculation is that the aPFC is triggering the sensory neocortex to render a specific simulation of the world. The aPFC first asks, “What happens if we go to the left?” The sensory neocortex then renders a simulation of turning left, which then passes back to the aPFC. The aPFC then says, “Okay, and then what happens if we keep going forward?” which the sensory neocortex renders again, and so on and so forth all the way to the imagined goal modeled in the aPFC.

Alternatively, it could be the basal ganglia that determines the actions taken during these simulations. This would be even closer to how AlphaZero worked—it selected simulated actions based on the actions its model-free actor predicted were best. In this case, it would be the aPFC that selects which of the divergent action predictions of the basal ganglia to simulate, but the basal ganglia would continue to decide which actions it wants to take in the imagined world rendered by the sensory neocortex.

Step #3: Choosing an Option

The neocortex simulates sequences of actions, but what makes the final decision as to what direction the rat will actually go? How does a rat choose? Here’s one speculation. The basal ganglia already has a system for making choices. Even the ancient vertebrates had to make choices when presented with competing stimuli. The basal ganglia accumulates votes for competing choices, with different populations of neurons representing each competing action ramping up in excitement until it passes a choice threshold, at which point

an action is selected.

And so, as the process of vicarious trial and error unfolds, the results of these vicarious replays of behavior accumulate votes for each choice in the basal ganglia—the same way it would if the trial and error were not vicarious but real. If the basal ganglia keeps getting more excited by imagining drinking water than by imagining eating food (as measured by the amount of dopamine released), then these votes for water will quickly pass the choice threshold. The basal ganglia will take over behavior, and the rat will go get water.

The emergent effect of all this is that the aPFC vicariously trained the basal ganglia that left was the better option. The basal ganglia doesn’t know whether the sensory neocortex is simulating the current world or an imagined world. All the basal ganglia knows is that when it turned left, it got reinforced. Hence, when the sensory neocortex goes back to simulating the current world at the beginning of the maze, the basal ganglia immediately tries to repeat the behavior that was just vicariously reinforced. Voilà—the animal runs to the left to get water.

Goals and Habits (or the Inner Duality of Mammals)

In the early 1980s, a Cambridge psychologist by the name of Tony Dickinson was engaging in the popular psychology experiments of the time: training animals to push levers to get rewards. Dickinson was asking a seemingly mundane question: What happens if you devalue the reward of a behavior after the behavior is learned? Suppose you teach a rat that pushing a lever releases a food pellet from a nearby contraption. The rat will rapidly go back and forth between pushing the lever and gobbling up the food. Now suppose one day, completely out of the context of the box with the lever, you give the rat the same pellet and secretly lace it with a chemical that makes the rat feel sick. How does this change their behavior?

The first result is expected: Rats, even after they have long recovered from their short bout of nausea, no longer find the pellets as appetizing as they did before. When given a mound of such pellets, rats eat substantially fewer. But the more interesting question was this: What would happen the next time the rats were shown the lever? If animals are simply governed by Thorndike’s law of effect, then they would run up to the lever and push it as rapidly as before—the lever had been reinforced many times, and nothing had yet unreinforced the act of pushing the lever. On the other hand, if animals are indeed able to simulate the consequences of pushing the lever and realize that the result is a pellet that they no longer like, they won’t want to push the lever as much. What Dickinson found was that after this procedure, rats who had the pellet paired with illness pushed the lever almost 50 percent less than

those that had not.

This is consistent with the idea that the neocortex enables even simple mammals such as rats to vicariously simulate future choices and change their behaviors based on the imagined consequences. But as Dickinson continued these experiments, he noticed something weird: Some rats continued to push the lever with as much, if not more, vigor after the pellet was paired with illness. Some rats became what he called “insensitive to devaluation.” The difference, he found, was merely a consequence of how many times the rats had pushed the lever to get a reward. Rats that had done the task one hundred times did the smart thing—they no longer wanted to push the lever once

the food was devalued. But rats that had done the task five hundred times ran up to the lever and just started pushing it like crazy, even if the food was devalued. And in all these tests, the food pellets never were given; the group that had become insensitive to devaluation just kept pushing the lever without ever getting a reward.

Dickinson had discovered habits. By engaging in the behavior five hundred times, rats had developed an automated motor response that was triggered by a sensory cue and completely detached from the higher-level goal of the behavior. The basal ganglia took over behavior without the aPFC pausing to consider what future these actions would produce. The behavior had been repeated so many times that the aPFC and basal ganglia did not detect any uncertainty and therefore the animal did not pause to consider the consequences.

Perhaps this is a familiar experience. People wake up and look at their phones without asking themselves why they are choosing to look at their phones. They keep scrolling through Instagram even though if someone had asked them if they wanted to keep scrolling, they’d say “no.” Of course, not all habits are bad: You don’t think about walking, and yet you walk perfectly; you don’t think about typing, and yet the thoughts flow effortlessly from your mind to your fingertips; you don’t think about speaking, and yet thoughts magically convert themselves into a repertoire of tongue, mouth, and throat movements.

Habits are automated actions triggered by stimuli directly (they are model-free). They are behaviors controlled directly by the basal ganglia. They are the way mammalian brains save time and energy, avoiding unnecessarily engaging in simulation and planning. When such automation occurs at the right times, it enables us to complete complex behaviors easily; when it occurs at the wrong times, we make bad choices.

The duality between model-based and model-free decision-making methods shows up in different forms across different fields. In AI, the terms model-based and model-free are used. In animal psychology, this same duality is described as goal-driven behavior and habitual behavior. And in behavioral economics, as in Daniel Kahneman’s famous book Thinking, Fast and Slow, this same duality is described as “system 2” (thinking slow) versus “system 1” (thinking fast). In all these cases, the duality is the same: Humans and, indeed, all mammals (and some other animals that independently evolved simulation) sometimes pause to simulate their options (model-based, goal-driven, system 2) and sometimes act automatically (model-free, habitual, system 1). Neither is better; each has its benefits and costs. Brains attempt to intelligently select when to do each, but brains do not always make this decision correctly, and this is the origin of many of our irrational behaviors.

The language used in animal psychology is revealing—one type of behavior is goal-driven and the other is not. Indeed, goals themselves may not have evolved until early mammals.

The Evolution of the First Goal

Just as the explanations of sensory information are not real (i.e., you don’t perceive what you see), so intent is not real; rather, it is a computational trick for making predictions about what an animal will do next.

This is important: The basal ganglia has no intent or goals. A model-free reinforcement learning system like the basal ganglia is intent-free; it is a system that simply learns to repeat behaviors that have previously been reinforced. This is not to say that such model-free systems are dumb or devoid of motivation; they can be incredibly intelligent and clever, and they can rapidly learn to produce behavior that maximizes the amount of reward. But these model-free systems do not have “goals” in the sense that they do not set out to pursue a specific outcome. This is one reason why model-free reinforcement learning systems are painfully hard to interpret—when we ask, “Why did the AI system do that?,” we are asking a question to which there is really no answer. Or at least, the answer will always be the same: because it thought that was the choice with the most predicted reward.

In contrast, the aPFC does have explicit goals—it wants to go to the fridge to eat strawberries or go to the water fountain to drink water. By simulating a future that terminates at some end result, the aPFC has an end state (a goal) that it seeks to achieve. This is why it is possible, at least in circumstances where people make aPFC-driven (goal-oriented, model-based, system 2) choices, to ask why a person did something.

It is somewhat magical that the very same neocortical microcircuit that constructs a model of external objects in the sensory cortex can be repurposed to construct goals and modify behavior to pursue these goals in the frontal cortex. Karl Friston of University College London—one of the pioneers of the idea that the neocortex implements a generative model—

calls this “active inference.” The sensory cortex engages in passive inference—merely explaining and predicting sensory input. The aPFC engages in active inference—explaining one’s own behavior and then using its predictions to actively change that behavior. By pausing to play out what the aPFC predicts will happen and thereby vicariously training the basal ganglia, the aPFC is repurposing the neocortical generative model for prediction to create volition.

When you pause and simulate different dinner options, choose to get pasta, then begin the long action sequence to get to the restaurant, this is a “volitional” choice—you can answer why you are getting in the car; you know the end state you are pursuing. In contrast, when you act only from habit, you have no answer as to why you did what you did.

Karl Friston also offers an explanation for the perplexing fact that some parts of the frontal cortex are

missing the fourth layer of the neocortical column. What does layer four do? In the sensory cortex, layer four is where raw sensory input flows into a neocortical column. Layer four is speculated to have the role of pushing the rest of the neocortical column to render a simulation that best matches its incoming sensory data (perception by inference). There is evidence that when a neocortical column is engaged in simulating, layer-four activity declines as active incoming sensory input is suppressed—this is how the neocortex can render a simulation of something not currently experienced (e.g., imagining a car when

looking at the sky). This is a clue. Active inference suggests that the aPFC constructs intent and then tries to predict behavior consistent with that intent; in other words, it tries to make its intent come true. If the animal does something inconsistent with the aPFC’s constructed intent, the aPFC doesn’t want to adjust its model of intent to match the behavior, it wants to adjust the behavior: if you are thirsty and your basal ganglia makes a decision to go in the direction that has no water, the aPFC doesn’t want to adjust its model of your intent to assume you are not thirsty, instead it wants to pause the basal ganglia’s mistake, and convince it to turn around and go toward the water. Thus, the aPFC spends very little, if any, time trying to match its inferred intent to the behavior it sees, and so it has no need for a large, or even any, layer four.

Of course, the aPFC isn’t evolutionarily programmed to understand the goals of the animal, instead it learns these goals by first modeling behavior originally controlled by the basal ganglia. The aPFC constructs goals by observing behavior that is originally entirely devoid of them. And only once these goals are learned does the aPFC begin to exert control over behavior: the basal ganglia begins as the teacher of the aPFC, but as a mammal develops, these roles flip, and the aPFC becomes the teacher of the basal ganglia. And indeed, during brain development, agranular parts of the frontal cortex begin with a layer four that then slowly atrophies and disappears during development, leaving layer four largely empty. Perhaps this is part of a developmental program for constructing a model of self, starting by matching one’s internal model to its observations (hence beginning with a layer 4), and then transitioning to pushing behavior to match one’s internal model (hence no need for a layer 4 anymore). Again we see a beautiful bootstrapping in evolution.

This also offers some insight into the experience of Damasio’s patient L. It makes some sense why her head was “empty”: She was unable to render an inner simulation. She had no thoughts. She had no will to respond to anything because her inner model of intent was gone, and without that, her mind could not set even the simplest of goals. And without goals, tragically, nothing mattered.

How Mammals Control Themselves: Attention, Working Memory, and Self-Control

In a typical neuroscience textbook, the four functions ascribed to the frontal neocortex are attention, working memory, executive control, and, as we have already seen, planning. The connecting theme of these functions has always been confusing; it seems odd that one structure would subserve all these distinct roles. But through the lens of evolution, it makes sense that these functions are all intimately related—they are all different applications of controlling the neocortical simulation.

Remember the ambiguous picture of a

duck or a rabbit? As you oscillate between perceiving a duck or a rabbit, it is your aPFC that is nudging your visual cortex back and forth between each interpretation. Your aPFC can trigger an internal simulation of ducks when you close your eyes, and your aPFC can use the same mechanism to trigger an internal simulation of ducks with your eyes open and you are looking at a picture that could be either a duck or a rabbit. In both cases, the aPFC is trying to invoke a simulation; the only difference is that with your eyes closed, the simulation is unconstrained, and with your eyes open it is constrained to be consistent with what you are seeing. The aPFC’s triggering of simulation is called imagination when it is unconstrained by current sensory input and attention when it is constrained by current sensory input. But in both cases, the aPFC is, in principle, doing the same thing.

What is the point of attention? When a mouse selects an action sequence after its imagined simulation, it must stick to its plan as it runs down its path. This is harder than it sounds. The imagined simulation will not have been perfect; the mouse will not have predicted each sight, smell, and contour of the environment that it will actually experience. This means that the vicarious learning that the basal ganglia experienced will differ from the actual experience as the plan unfolds, and therefore, the basal ganglia may not correctly fulfill the intended behavior.

One way the aPFC can solve this problem is using attention. Suppose a rat’s basal ganglia learned, through trial and error, to run away from ducks and run toward rabbits. In this case, the basal ganglia will have opposite reactions to seeing the duck/rabbit depending on what pattern gets sent to it from the neocortex. If the aPFC had previously imagined seeing a rabbit and running toward it, then it can control the basal ganglia’s choices by using attention to ensure that when the rat sees this ambiguous picture, it sees a rabbit, not a duck.

Controlling ongoing behavior often also requires working memory—the maintenance of representations in the absence of any sensory cues. Many imagined paths and tasks involve waiting. For example, when a rodent forages among trees for nuts, it must remember which trees it has already foraged. This is a task shown to require the aPFC. If you inhibit a rodent’s aPFC during these delay periods, rodents lose their ability to perform such tasks from memory. And during such tasks, the aPFC exhibits “delay activity,” remaining activated even in the absence of any external cues. These tasks require the aPFC because working memory functions in the same way as attention and planning—it is the invoking of an inner simulation. Working memory—holding something in your head—is just your aPFC trying to keep re-invoking an inner simulation until you no longer need it.

In addition to planning, attention, and working memory, the aPFC can also control ongoing behavior more directly: It can inhibit the amygdala. There is a projection from the aPFC to inhibitory neurons surrounding the amygdala. During the fulfillment of an imagined plan, the aPFC can attempt to prevent the amygdala from triggering its own approach and avoidance responses. This was the evolutionary beginning of what psychologists call behavioral inhibition, willpower, and self-control: the persistent tension between our moment-to-moment cravings (as controlled by the amygdala and basal ganglia) and what we know to be a better choice (as controlled by the aPFC). In moments of willpower, you can inhibit your amygdala-driven cravings. In moments of weakness, the amygdala wins. This is why people become more impulsive when tired or stressed—the aPFC is energetically expensive to run, so if you are tired or stressed, the aPFC will be much less effective at inhibiting the amygdala.

To summarize: Planning, attention, and working memory are all controlled by the aPFC because all three are, in principle, the same thing. They are all different manifestations of brains trying to select what simulation to render. How does the aPFC “control” behavior? The idea presented here is that it doesn’t control behavior per se; it tries to convince the basal ganglia of the right choice by vicariously showing it that one choice is better and by filtering what information makes it to the basal ganglia. The aPFC controls behavior not by telling but by showing.

The benefit of this can be seen when comparing the performance of mammals to other vertebrates like lizards on tasks that require inhibiting reflexive responses in favor of “smarter” choices. If you put a lizard in a maze and try to train it to go toward a red light to get appealing food and avoid a green light which offers unappealing food, it takes

lizards hundreds of trials to learn this simple task. The hardwired preference of lizards toward green light takes a long time to be untrained. Without a neocortex to pause and vicariously consider options, the only way lizards learn this task is through endless real trial and error. In contrast, rats learn to inhibit their hardwired responses much more rapidly, an advantage that disappears

if you damage a rat’s aPFC.

Early mammals had the ability to vicariously explore their inner model of the world, make choices based on imagined outcomes, and stick to the imagined plan once chosen. They could flexibly determine when to simulate futures and when to use habits; and they intelligently selected what to simulate, overcoming the search problem. They were our first ancestors to have goals.

cortex damage also has this effect. In primates, neurons from the motor cortex send a projection directly to the spinal cord to control movement. All this leads to the conclusion that the motor cortex is the locus of motor commands; it is the controller of movement.

But there are three problems with this idea. First, the neocortical columns in the motor cortex have the same microcircuitry as

other areas of the neocortex. If we believe the neocortex implements a generative model that tries to explain its inputs and uses these explanations to make predictions, then we must account for how this could have been repurposed to produce motor commands.

Second, some mammals don’t have a motor cortex, and they can clearly move around normally. As discussed in the

The third problem with the “motor cortex equals motor commands” idea is that the paralysis caused by motor cortex damage is unique to primates; most mammals with motor cortex damage do not

suffer from such paralysis. Rats and cats with damaged motor cortices can still walk, hunt, eat, and move around just fine. The motor cortex was clearly not the locus of motor commands in early mammals, and it was only later—in primates—that it became required for movement. So why did the motor cortex evolve? What was its original function? What changed with primates?

Predictions, Not Commands

Karl Friston, the pioneer of the theory of active inference, offers an alternative interpretation of the motor cortex. While the prevailing view has always been that the motor cortex generates motor commands, telling muscles exactly what to do, Friston flips this idea on its head: Perhaps the motor cortex doesn’t generate motor commands but rather motor predictions. Perhaps the motor cortex is in a constant state of observing the body movements that occur in the nearby somatosensory cortex (hence why there is such an elegant mirror of motor cortex and somatosensory cortex) and then tries to explain the behavior and use these explanations to predict what an animal will do next. And perhaps the wiring is merely tweaked so that motor cortex predictions flow to the spinal cord and control our movement—in other words, the motor cortex is wired to make its predictions come true.

By this account, the motor cortex operates the same way the agranular prefrontal cortex operates. The difference is that the aPFC learns to predict movements of navigational paths, whereas the motor cortex learns to predict movements of specific body parts. The aPFC will predict that an animal will turn left; the motor cortex will predict that the animal will place its left paw exactly on a platform.

This is the general idea of “embodiment”—parts of the neocortex, such as the motor cortex and somatosensory cortex, have an entire model of an animal’s body that can be simulated, manipulated, and adjusted as time unfolds. Friston’s idea explains how the neocortical microcircuit could be repurposed to produce specific body movements.

But if most mammals can move around normally with no motor cortex, then what was its original function? If the aPFC enables the planning of navigational routes, what did the motor cortex enable?

Damage to the motor cortex in nonprimate mammals such as rodents and cats has two effects. First, animals become impaired at performing skilled movements, such as carefully placing a paw on a thin branch, reaching through a small hole to grasp a morsel of food, stepping over an obstacle once it is out of sight, or placing a foot on a

small unevenly placed platform. Second, nonprimate mammals become impaired at learning new sequences of movements that they have never performed before. For example, a rat trained to perform a specifically orchestrated sequence of movements can perform this sequence only if its motor cortex was damaged after the task was already well learned. If you damage a rat’s motor cortex before it is trained on this task, it becomes unable to

learn the lever sequence.

This suggests that the motor cortex was originally not the locus of motor commands but of motor planning. When an animal must perform careful movements—placing a paw on a small platform or stepping over an out-of-sight obstacle—it must mentally plan and simulate its body movements ahead of time. This explains why the motor cortex is necessary for learning new complex movements but not for executing well-learned ones. When an animal is learning a new movement, the motor cortex simulations vicariously train the basal ganglia. Once a movement is well learned, the motor cortex is no longer needed.

Recording activity in the motor cortex supports this idea. In nonprimate mammals, the motor cortex is most activated not by movement in general but specifically by

movements that require planning. Consistent with the idea that animals are simulating movements, the motor cortex and somatosensory cortex get activated in advance of an upcoming precision movement, even if obstacles are not seen but simply

known to be there. And this activity maintains itself until the animal has completed the

movement it presumably planned.

In humans there is plenty of evidence that the premotor and motor cortices are activated both by doing movements and by imagining movements: For example, have someone think about walking, and the leg area of the

motor cortex becomes activated. This intertwining of the neurological infrastructure for imagined movements and actual movements can be observed not only in brain recordings, but also in physical lab experiments. Sit a human down in a chair and ask them to do nothing more than simply maintain their upright posture. Then play audio recordings of arbitrary sentences. Her posture gets worse when she hears sentences like “I get up, put on my slippers, go to the bathroom,” but not when she hears sentences

unrelated to movement. Just hearing those sentences activated an inner simulation of a changing posture, which affected her actual posture. Of course, this inner simulation also comes with benefits (it doesn’t just mess up our posture): Mental rehearsal of motor skills substantially increases performance across speaking, golf swings,

and even surgical maneuvers.

The motor cortex’s skill in sensorimotor planning enabled early mammals to learn and execute precise movements. When comparing the motor skills of mammals to that of reptiles, it is quite clear that mammals are uniquely capable when it comes to fine motor skills. Mice can pick seeds up and skillfully break them open. Mice, squirrels, and cats are incredibly skillful tree climbers, effortlessly placing their limbs in precise places to ensure they don’t fall. Squirrels and cats can plan and execute extremely accurate and precise jumps across platforms. If you have ever had a pet lizard or a turtle, you know that such skills are not the province of most reptiles. In fact, studies that examined lizards running over obstacles revealed how surprisingly sloppy

the whole affair is. They don’t anticipate obstacles or modify their forelimb placements

to get around platforms. Given their seeming inability to plan movements in advance, it is perhaps unsurprising that few reptiles live in trees and that those that do move slowly, in contrast to the rapid skillful running and jumping of arboreal mammals.

A Hierarchy of Goals: A Balance of Simulation and Automation

How does all this work together? The frontal neocortex of early placental mammals was organized into a hierarchy. At the top of the hierarchy was the agranular prefrontal cortex, where high-level goals are constructed based on amygdala and hypothalamus activation. The aPFC might generate an intent like “drink water” or “eat food.” The aPFC then propagates these goals to a nearby frontal region (the premotor cortex), which constructs subgoals and propagates these subgoals further until they reach the motor cortex, which then constructs sub-subgoals. The intent modeled in the motor cortex are these sub-subgoals, which can be as simple as “Position my index finger here and my thumb over here.”

This hierarchy enables more efficient processing by distributing effort across many different neocortical columns. The aPFC doesn’t have to worry about the specific movements necessary to achieve its goals; it must worry only about high-level navigational paths. Similarly, this allows the motor cortex not to have to worry about the high-level goal of the behavior and worry only about accomplishing specific low-level movement goals (picking up a cup or playing a specific chord).

The basal ganglia makes loops of connectivity with the frontal cortex, with the aPFC connecting to the front region of the basal ganglia (which then connects back to aPFC through the thalamus), and the motor cortex connecting to the back region of the basal ganglia (which then connects back to the motor cortex through a different region of the thalamus). These loops are so elegantly and particularly wired that it is hard to resist trying to reverse-engineer what they are doing.

The leading view among neuroscientists is that these are subsystems designed to manage different levels of the motor hierarchy. The front part of the basal ganglia automatically associates stimuli with high-level goals. It is what generates cravings: You come home and smell rigatoni, and suddenly you are on a mission to eat some. Drug addicts show extreme activations of this front part of the basal ganglia when they see stimuli that create drug cravings. The aPFC, however, is what makes you pause and consider if you actually want to pursue these cravings (“What about our diet?”). The back part of the basal ganglia automatically associates stimuli with low-level goals, such as specific body movements. It is what generates automatic skilled movements. The motor cortex, on the other hand, is what makes you pause and plan out your exact movements ahead of time.

	HIGH-LEVEL GOALS	LOW-LEVEL GOALS
SIMULATION	Agranular Prefrontal Cortex Simulates navigational paths Asks “do I want rigatoni or would I rather diet?” Damage causes impairments in planning navigational routes	Motor Cortex Simulates body movements Asks “how do I configure my fingers to play this C chord that I just learned on guitar?” Damage causes impairments in learning new motor skills and executing fine motor skills
AUTOMATION	Front part of basal ganglia Automatic pursuit of a high-level goal in response to stimulus Produces habitual cravings Damage can cure drug addiction	Back part of basal ganglia Automatic execution of motor skill in response to stimulus Produces habitual motor responses Damage causes impairments in executing learned skills, and impairs motor habit formation

Any level of goal, whether high-level or low-level goals, has both a self model in the frontal neocortex and a model-free system in the basal ganglia. The neocortex offers a slower but more flexible system for training, and the basal ganglia offers a faster but less flexible version for well-trained paths and movements.

There is plenty of evidence for such a motor hierarchy. Recordings have shown that neurons in the aPFC are sensitive to high-level goals, whereas those in the premotor and motor cortex are sensitive to progressively

lower-level subgoals. Learning a new behavior activates all levels of the motor hierarchy at first, but as the behavior becomes automatic, it activates only lower

levels in the hierarchy. If you damage high-level parts of the motor hierarchy (aPFC or the front part of the basal ganglia) in rats, it makes them less sensitive to high-level goals (they will keep pushing a lever even though they no longer want the food it produces). In contrast, if you damage low-level parts of the motor hierarchy, rats become more sensitive to high-level goals, and they struggle to create motor habits (e.g., rats won’t develop a habit for pushing a lever no matter how many

trials they go through).

While damage to the aPFC deprives animals of their intentions, as we saw with patient L, damage to parts of the premotor cortex seems to disconnect the proper flow of these intentions from high-level goals to specific movements. This can cause “alien limb syndrome”: patients will claim that certain parts of their bodies are moving on their

own without their control. Signs of such alien movements are also seen in rodents with

damage to the premotor cortex. Such damage also causes what is called “utilization behavior” or “field-dependent behavior,” where patients will execute motor sequences devoid of any clear goal: They will drink from empty cups, put on other people’s jackets even though they aren’t going anywhere, scribble with pencils, or do any other behavior

that nearby stimuli suggest. All this is the result of a broken hierarchy—parts of the motor cortex are now unconstrained by top-down intentions from the aPFC flowing through the premotor cortex, and hence the motor cortex independently sets low-level goals for motor sequences.

There is also plenty of evidence for the idea that the frontal neocortex is the locus of simulation, while the basal ganglia is the locus of automation. Damaging an animal’s motor cortex impairs movement planning and learning new movements but not the execution of well-trained movements (because the back part of the basal ganglia already learned them). Similarly, damaging an animal’s aPFC impairs path planning and learning new paths but not the execution of well-trained paths.

Further, the front part of the basal ganglia demonstrates all the signs of an automatic selection of cues to pursue (i.e., automated high-level behaviors). When you see a cue that creates a craving, the part of the brain that is most activated is this front part of the basal ganglia. Those who try to inhibit their cravings show additional activation of frontal areas like the aPFC (simulating the negative consequences and trying to train the basal ganglia to make the harder choice). In fact, lesioning the front part of the basal ganglia is an effective (although highly controversial and questionably ethical) treatment for drug addiction. The relapse rate for heroin addicts is absurdly high; some estimate as high as 90 percent. One study in China took the most severe heroin addicts and lesioned the front part of their basal ganglia. The relapse

rate dropped to 42 percent. People lose the automatic behavior of pursuing cues and generating cravings (of course, there are also many side effects of such a surgery).

An intact and well-functioning motor hierarchy would have made the behavior of early placental mammals impressively flexible; animals could set high-level goals in the aPFC while lower-level areas of the motor hierarchy could flexibly respond to whatever obstacles present themselves. A mammal pursuing faraway water could continuously update its subgoals as events unfolded—the premotor cortex could respond to surprising obstacles by selected new movement sequences, and the motor cortex could adjust even the subtlest of specific movements of limbs, all in the name of a common goal.

The secret to dishwashing robots lives somewhere in the motor cortex and the broader motor system of mammals. Just as we do not yet understand how the neocortical microcircuit renders an accurate simulation of sensory input, we also do not yet understand how the motor cortex simulates and plans fine body movements with such flexibility and accuracy and how it continuously learns as it goes.

But if we use the past few decades as our guide, roboticists and AI researchers will likely figure this out, perhaps in the near future. Indeed, robotics are improving at a rapid pace. Twenty years ago, we could barely get a four-legged robot to balance itself upright, and now we have humanoid robots that can do flips in the air.

If we successfully build robots with motor systems similar to those of mammals, they will come along with many desirable properties. These robots will automatically learn new complex skills on their own. They will adjust their movements in real time to account for perturbations and changes in the world. We will give them high-level goals, and they will be able to figure out all the subgoals necessary to achieve it. When they try to learn some new task, they will be slow and careful as they simulate each body movement before they act, but as they get better, the behavior will become more automatic. Over the course of their lifetimes, the speed with which they learn new skills will increase as they reapply previously learned low-level skills to newly experienced higher-level goals. And if their brains work at all like mammal brains, they will not require massive supercomputers to accomplish these tasks. Indeed, the entire human brain operates on about the same amount of energy as a lightbulb.

Or maybe not. Perhaps roboticists will get all this to work in a very nonmammalian way—perhaps roboticists will figure it all out without reverse-engineering human brains. But just as bird wings were an existence proof for the possibility of flight—a goal for humans to strive for—the motor skills of mammals are our existence proof for the type of motor skills we hope to build into machines one day, and the motor cortex and the surrounding motor hierarchy are nature’s clues about how to make it all work.

of land-living vertebrates. This was the Permian-Triassic extinction.

Many of the other extinction events in the history of Earth seem to have been self-imposed—the Great Oxygenation Event was caused by cyanobacteria, and the Late Devonian Extinction was possibly caused by overproliferation of plants on land. But this one was not a fault of life but a fluke of an ambivalent universe.

Eventually, after over a hundred years, the blackened clouds began to fade. As the sun reemerged, plants began recovering lost ground and refilling the parched dead land. But this was a new world. Almost every dinosaur species was extinct except for one: the birds. Although our squirrel-like ancestor could not have known it and had not lived to see it, her offspring would inherit this new Earth. As the Earth healed, these small mammals found themselves in a completely new ecological playground. Without their dinosaur predators, they were free to explore new ecological niches, to diversify into new shapes and sizes, to conquer new territories, and to find new footing within the food chain.

The era that followed has been called the Era of Mammals. Descendants of these early mammals would eventually evolve into modern-day horses, elephants, tigers, and mice. Some would even reenter the sea and become today’s whales, dolphins, and seals. Some would take to the sky and become today’s bats.

Our direct ancestors were the ones who found refuge in the tall trees of Africa. These were some of the first primates. They shifted from being night-living (nocturnal) to being day-living (diurnal). As they became larger, they developed opposable thumbs to grasp branches and hold their heavier bodies. To support their bigger size, they shifted from an insect-based diet to a fruit-based diet. They lived in groups, and as they grew, they became relatively free from predation and food competition. And most notably, their brains exploded to well over a hundred times their original size.

Many lineages of mammals went on to have brains not much larger (proportionally) than those of early mammals, it was only in certain lineages of mammals, such as those of elephants, dolphins, and primates, where brains dramatically expanded. Because this book is about the human story, we will focus on the journey by which the primate brain became big. Indeed, why primates have such big brains—and specifically such large neocortices—is a question that has perplexed scientists since the days of Darwin. What was it about the lifestyle of early primates that necessitated such a big brain?

The Social-Brain Hypothesis

In the 1980s and 1990s, numerous primatologists and evolutionary psychologists, including Nicholas Humphrey, Frans de Waal, and Robin Dunbar, began speculating that the growth of the primate brain had nothing to do with the ecological demands of being a monkey in the African jungles ten to thirty million years ago and was instead a consequence of the unique social demands. They argued that these primates had stable mini-societies: Groups of individuals that stuck together for long periods. Scientists hypothesized that to maintain these uniquely large social groups, these individuals needed unique cognitive abilities. This created pressure, they argued, for bigger brains.

A simple test of this theory would be to look at the mini-societies of monkeys and apes around the world and see if the size of their neocortices relative to the rest of their brains correlated with their social-group size. It was Robin Dunbar who did this, and what he found shook the field. This correlation has been confirmed across many primates: the bigger the neocortex of a primate, the

bigger its social group.

But monkeys and apes are far from the only mammals, let alone animals, that live in groups. And interestingly, this correlation does not hold

for most other animals. The brain of a buffalo living in a thousand-member herd is not meaningfully bigger than the brain of a solitary moose. It isn’t group size in general but the specific type of group that early primates created that seemed to have required larger brains. There was something unique about primate groups relative to those of most other mammals, something we can understand only by understanding the general drive for grouping itself.

The Evolutionary Tension Between the Collective and the Individual

Early mammals were likely more social than the amniotes (their lizard-like ancestors) that came before. These early mammals uniquely gave birth to helpless children. This dynamic would have been tenable only if mothers built a strong bond to help, nurture, and physically protect their children. Further, mammals engage in play much more than other vertebrates. Even the offspring of simple mammals like rats play with each other, play mounting and play fighting. These early acts of play might have served the purpose of refining and training the motor cortex of young mammals so that in higher-stake situations they wouldn’t be learning from scratch. In these early mammals, this collaborative period between mother and child was relatively short-lived. After a period of childhood development, the bond tends to fade, and the children and mothers go their separate ways. This is how it is for many mammals that spend most of their lives on their own, like tigers and bears.

But not all animals separate in adulthood like this. In fact, the simplest, most widely used, and likely first collective behavior in animals was group living, whereby animals of the same species simply clustered together. Fish reflexively follow each other’s movements and swim near one another. Many herbivorous dinosaurs lived in herds. And, of course, this is seen across mammals as well—buffalo and antelope live in herds. The key benefit of group living is that it helps stave off predators. If even a single antelope in a herd catches a glimpse of a nearby lion and begins running away, it cues all the others in the herd to follow. While a lone antelope is easy prey, a herd of them can be dangerous to even a lion.

However, group living is not a freely gained survival benefit—it comes at a high cost. In the presence of food constraints or limited numbers of eligible mates, a herd of animals creates dangerous competition. If this competition leads to infighting and violence, the group ends up wasting precious energy competing and fighting each other. In such a circumstance, the same number of animals would have been better off living separately.

Thus, animals who fell into the strategy of group living evolved tools to resolve disputes while minimizing the energetic cost of such disputes. This led to the development of mechanisms to signal strength and submission without having to actually engage in a physical altercation. Deer and antelope lock horns to compete for food and mates, a much cheaper form of competition than fighting. Bears, monkeys, and dogs bare their teeth and growl to show aggression.

These animals also evolved mechanisms to signal submission, allowing them to acknowledge defeat and show that it is unnecessary for others to spend energy hurting them. Dogs bow and roll over to their backs; bears sit

down and look away; and deer lower their heads

and flatten their ears. All of this provided a mechanism to ameliorate tensions and lower the amount of energy spent infighting.

With the ability to signal strength and submission, many animals were able to make group living work. Most lineages of mammals fell into one of four buckets of social systems: solitary, pair-bonded, harems, and

multi-male groups. The solitary mammals, such as moose, spend a large portion of their adult lives on their own, meeting up primarily for mating and, if female, raising children. The pair-bonding mammals, such as red foxes and prairie voles, live together in pairs, raising children together. Other mammals, such as camels, live in harems, which are social groups with a single dominant male and many females. And then there are mammals that live in multi-male groups, social groups with many males and many females living together.

While solitary and pair-bonded mammals avoid the downsides of large social groups, they also miss out on the benefits. On the other hand, harems and multi-male groups reap the benefits of larger groups, but incur the costs of competition. In addition to displays of aggression and submission, another way harems and multi-male groups minimize competition is through hierarchical rigidity. In harems, there is a single dominant male who does all of the mating; the only other males allowed in the group are

his own children.

SOLITARY	PAIR-BONDING	HAREMS	MULTI-MALE GROUPS
Independent	One male, one female	One male, many females	Many males, many females
Live mostly independently in adulthood	One male and one female live and raise children together.	A single dominant male living with a group of females who have their own hierarchy.	Separate male and female hierarchy
Tigers Jaguar Moose	Red foxes Prairie voles Giant otters Pygmy marmoset	Mongolian camels Fur seals Gorillas	Lions Hippopotamuses Lemurs Chimpanzees Baboons Macaque monkeys

Multi-male groups also work via hierarchical rigidity: There is a strict hierarchy of both males and females. Low-ranking males are allowed in the group, but they do little mating and get the last pick of food; the high-ranking males eat food first and do most, if not all, of the mating.

How is the hierarchy decided in these social groups? It’s simple—the strongest, biggest, and toughest become dominant. The locking of horns and baring of teeth are all designed to demonstrate who would win in a fight while avoiding the fight itself.

Early primates made this same evolutionary trade-off—they evolved to live in groups, accepting the risk of aggression for the benefit of the better predator

avoidance of large groups. The greater the predation risk experienced by primates, the larger the social group

they created in response. Like many modern primates, these early primates probably lived in multi-male groups with hierarchies of males and females where lower-ranking members got the worst food and almost never mated and high-ranking members got their pick of each. They bared their teeth as a show of aggression. We know this from fossils and through observing the behavior across many of today’s monkey and ape societies: chimpanzees, bonobos, and macaques all live this way. At first glance, these early primate groups would have looked no different from other mammals’ multi-male groups. But as researchers observed monkey and ape behavior more thoroughly, it became clear that when it comes to sociality, primates are unlike most of their mammalian cousins. Something happened in the sociality of early primates that was different from the sociality that had evolved before in early mammals.

living in a one-acre forest. Inspired by Tolman’s experiments on mental maps in rats, Menzel was experimenting with the mental maps of chimpanzees. His main interest was in whether chimpanzees could remember the locations of hidden morsels of food.

Menzel would hide some food in a random location within this one-acre area, perhaps under a rock or within a bush, and then reveal its location to one of the chimpanzees. He would then place food back in these locations on a recurring basis. Chimpanzees, like rats, were eminently able to remember these exact locations, learning to recheck these specific spots to forage Menzel’s hidden food. But Menzel began spotting behavior that was quite unlike that of a rat, behavior that he had never intended to investigate and had never expected to find. Indeed, while merely investigating spatial memory, Menzel unearthed behavior that was eerily Machiavellian.

When Menzel first revealed the location of hidden food to one of the subordinate chimpanzees, named Belle, she happily alerted the rest of the group and shared the food. But when the dominate male, Rock, came over to indulge in the treat, he took all the food for himself. After Rock did this a few times, Belle stopped sharing, and began engaging in ever more sophisticated strategies to withhold information about the hidden locations of food from Rock.

At first, Belle simply sat on top of the secret location of the food to hide it from Rock, and only when he was far away would she uncover and openly eat the food. But when Rock realized she was hiding the food underneath her, he began pushing her to get at the food. In response to this, Belle came up with a new strategy—once she was shown a new location for the hidden food, she didn’t go to it immediately. She would wait for Rock to look away, and then she would run to the food. In response to this new strategy, Rock began trying to trick Belle: He would look away and act uninterested, and once Belle went for the food, he would turn around and run toward it. Belle even began trying to lead Rock in the wrong directions, a deception Rock eventually realized and thus, in response, began to search for food in the opposite direction that Belle would try to lead him.

This process of ever-escalating deceptions and counter-deceptions reveals that both Rock and Belle were able to understand the other’s intent (“Belle is trying to lead me away from the food,” “Rock is trying to trick me by looking away”), as well as understand that it is possible to manipulate the other’s beliefs (“I can make Belle think I am not looking by pretending to be disinterested,” “I can make Rock think the food is in the wrong location by leading him in that direction”). Since Menzel’s work, numerous other experiments have similarly found that apes can, in fact, understand the intentions of others. Consider the following study: Apes were tested on their ability to tell the difference between

“accidental” and “intentional” actions. Show chimpanzees or orangutans three boxes, one of which has food in it. The box filled with food can be differentiated from the others because it has a thick mark from a pen on it. Do this a few times until they learn that food is always in the box with the mark on it. Then have an experimenter come in with the three boxes and intentionally mark one box by leaning over to mark it and “accidently” mark another box by passively dropping the marker on it. When the apes are allowed to come to the boxes to look for food, which box do they go to? They immediately go for the box the experimenter “intentionally” marked and ignore the one that was “accidently” marked. Apes had deduced the intent of the experimenter.

Consider another study. Have a chimpanzee sit across from two experimenters who have food near them. One experimenter is unable to give food for various reasons (sometimes unable to see the food; sometimes the food is stuck; other times they demonstrate losing the food). Another experimenter is unwilling to give them food (simply having food but not giving it). Neither experimenter provided food, and yet chimpanzees treat these two cases differently. When given the chance to pick between these two experimenters, chimps always went back to the person who seemed unable to help and avoided

those who seemed unwilling. Chimpanzees seemed able to use cues about the situation of another person (Can they see the food? Did they lose it? Can they just not get it?) to reason about their intentions, and thereby predict the likelihood this other person would give them food in the future.

Understanding the minds of others requires understanding not only their intentions but also their knowledge. Belle sitting on the food to hide it from Rock was Belle’s attempt to manipulate Rock’s knowledge. In another test, chimps were given the chance to play with two sets of goggles—one was transparent and easy to see through, and the other was opaque and hard to see through. When given the chance to ask for food from human experimenters wearing those same goggles, chimps knew to go to the human that was wearing the transparent goggles—they could tell the human wearing the opaque

goggles wouldn’t see them.

The degree to which animals can infer the intent and knowledge of others continues to be controversial in animal psychology. While there is meaningful evidence that many primates (especially apes) have this ability, the evidence in other animals is less clear. It is possible that other intelligent animals,

like some birds, dolphins, and dogs, also can do this. My argument is not that only primates can do this but that this ability was not present in early mammals, and in the human lineage this feat emerged with early primates (or at least by early apes). Even dogs, as socially intelligent and attentive to humans as they are, may be unable to understand that humans can hold different knowledge. Allow a dog to see its trainer put a treat in one location and then allow the dog to see someone else put a treat in a different location (while the trainer is not present and hence unaware of this other treat). When the trainer comes back and commands, “Get the treat,” dogs are equally likely to run to either location, failing to identify the location the trainer is referring to based on which

location the trainer knows about.

This act of inferring someone’s intent and knowledge is called “theory of mind”—so named because it requires us to have a theory about the minds of others. It is a cognitive feat that evidence suggests emerged in early primates. And as we will see, theory of mind might explain why primates have such big brains and why their brain size correlates with group size.

Primate Politics

The most obvious social behavior of nonhuman primates is grooming—a pair of monkeys will take turns picking dirt and mites from each other’s backs where they can’t reach themselves. In the first half of the twentieth century, this behavior was believed to be primarily for hygienic purposes. But it is now undisputed that this grooming behavior serves more of a social purpose than a hygienic purpose. There is no correlation between time spent grooming and body size (which you would expect if the function of grooming was for cleaning the body), but there is a strong correlation between time spent

grooming and group size. Further, individuals that don’t get groomed by others a lot don’t make up for it

by grooming themselves more. And individual monkeys have very specific grooming partners that persist over long periods, even lifetimes.

Primate groups are not made up of a social mash of randomly interacting individuals; these mini-societies of fifteen to fifty primates are made up of subnetworks of dynamic and specific relationships. Monkeys keep track of and remember each individual in their group and are able to recognize them

by appearance and voice. They keep track not only of individuals but also of the specific relationships between individuals. When a faraway distress call is heard from a child, individuals of the group all immediately look not in the direction of the distress call but at the mother of the distressed child—Oh no, what will Alice do to help her daughter? Or Can we trust this kid? Let’s see

The relationships between individuals are not only familial but also hierarchical. Vervet monkeys have an approach-retreat routine to signal dominance and submission; when a higher-ranking individual walks toward a lower-ranking individual, the lower-ranking individual retreats. These dominance relationships persist across contexts: When monkey A expresses submission to B in one situation, it is almost always the case that A will also submit to B in another situation. These dominance relationships are transitive: If you see monkey A submit to B, and B submit to C, then it is almost definitely the case that

A will submit to C. And these hierarchies often persist for

many years, even generations. These signals of dominance and submission are not one-off displays; they represent an explicit social hierarchy.

Primates are extremely sensitive to interactions that violate the social hierarchy. In a study done in 2003, experimenters obtained audio recordings of different members of a group of baboons making dominance or submission noises, and then set up speakers near the baboons so they could

play back these recordings. When they played a recording of a higher-ranking baboon making a dominance noise followed by a recording of a lower-ranking baboon making a submission noise, no baboons looked over to the speakers—nothing surprising about someone establishing their dominance over someone ranked below them. However, when they played a recording of any lower-ranking baboon making a dominance noise followed by a recording of a baboon ranked above them making a submission noise—a violation of the hierarchy—baboons freaked out and stared at the speakers to see what in the world had just happened. Like when a nerd smacks the popular bully in the face, the class can’t help but gawk: Did that really just happen?

What makes these monkey societies unique is not the presence of a social hierarchy (many animal groups have social hierarchies), but how the hierarchy is constructed. If you examined the social hierarchy of different monkey groups, you would notice that it often isn’t the strongest, biggest, or most aggressive monkey who sits at the top. Unlike most other social animals, for primates, it is not only physical power that determines one’s social ranking but also political power.

As was the case in many early human civilizations (and unfortunately still many today), one thing that determines a monkey’s place in its group is the family it is born into. In primate social groups, there tends to be a hierarchy of families. Here is a common structure for female hierarchies: At the top of the hierarchy is the eldest female of the highest-ranking family, followed by her offspring, then the eldest member of the second-highest-ranking family, followed by her offspring, and so

on and so forth. And when a daughter’s mother dies, she tends to inherit the

rank of her mother.

In a clear departure from the typical nonprimate association of strength with rank, a weak and frail juvenile member of a powerful family can easily scare off a much larger and stronger adult monkey from a family that ranks below her. In fact, the child herself is clearly aware of her place in the social structure—even young children will regularly challenge adults of lower-ranking families, but they won’t challenge adults

of higher-ranking families.

And as it goes with human societies—with endless oscillations of dynastic power struggles of families rising and collapsing—monkey dynasties rise and fall too. Families face incredible pressure to improve their position. Higher-ranking monkeys get their pick of food, grooming partners, mates, and resting sites. One’s evolutionary fitness improves with one’s rank; higher-ranking monkeys have more children and are less likely

to die from disease. And so if a high-ranking family sufficiently dwindles in number, a lower-ranking family will wage a coordinated mutiny; the lower-ranking family will make persistent aggressive challenges until the higher-ranking family submits, at which point a new

hierarchy has been established.

Such mutinies are not inevitable; high-ranking families that are few in number can form alliances with members outside of their families to help solidify their position. Indeed, about 20 percent of the time that an aggressive display is made, other nearby monkeys will respond by joining forces with either the attacker or the defender. Most of the time it is family members that come to help, but about a third of the time it is

nonfamily members that come. It seems that the ability to forge such allyships is one of the primary determinants of an individual’s rank: higher-ranking monkeys tend to be better at recruiting allies from unrelated individuals, and hierarchy reversals most often happen when monkeys fail

to recruit such allies.

Monkey politics happens in the dynamics of these allyships, which are forged not through fixed familial relationships but grooming and supporting others in conflicts. Allyships and grooming partnerships represent a common relationship, what we would call a friendship: monkeys most often rescue those whom they have previously

formed grooming partnerships with. And a monkey can earn reciprocity through acts of kindness even to those they are not currently friends with. If monkey A goes out of its way to groom monkey B, then B is substantially more likely to run to A’s defense next time A

makes a “help me” vocalization. This is also true with going out of one’s way to support others in conflicts—monkeys tend to run to the defense of those who have run

to their own defense. In these allyships is also a notion of trust: When a chimpanzee is given the option of (a) an experimenter giving her a mediocre snack directly, or (b) an experimenter giving another chimp, who will hopefully share, an amazing snack, chimps chose (b) only if the other chimp is a grooming partner. Otherwise they took the worse

food just for themselves.

These allyships have a big impact on a monkey’s political standing and quality of life. Those with power benefit from forging a sufficient coalition of lower-ranking allies, and low-ranking monkeys can substantially improve their lives by forging friendships with the right high-ranking families. Low-ranking individuals with powerful grooming partners get harassed much less, even when the high-ranking ally is out of sight; everyone in the group knows Don’t mess with James unless you want

High-ranking monkeys are more tolerant of the low-ranking individuals with whom they have forged allyships, giving them

more access to food.

Much of monkey social behavior suggests an incredible degree of political forethought. Monkeys prefer to invest in relationships with those

ranked higher than themselves. Monkeys prefer to mate with higher-ranking

members of the group. Monkeys compete to groom

with high-ranking individuals. When a dispute breaks out, monkeys are biased toward joining forces with

the higher-ranking individual. The children of high-ranking mothers are

the most popular playmates.

High-ranking monkeys also exhibit a cleverness in which lower-ranking members they choose to befriend. In a study where different low-ranking monkeys were trained to do specific tasks to obtain food, high-ranking monkeys quickly befriended those who had specialized skills, and they persisted in these grooming partnerships even after there was no immediate prospect of gaining food: I can see you are useful; let me take

.

Monkeys also show political cleverness in the aftermath of conflict. They go out of their way to try and “make up” after aggressive interactions, especially those

with nonfamily members. They often seek to hug and groom those they fought with, and they also try to make up with the individuals’ families, spending double the usual amount of time with family members of those they

have recently quarreled with.

The Arms Race for Political Savvy

Somehow the evolutionary trajectory of early primates led to the development of the incredibly broad suite of complex social behaviors that are seen across modern species of primates. And within these behaviors we see hints of the behavioral foundation of how humans tend to interact with each other. Why primates evolved these instincts is not exactly clear, but it may have had to do with the unique niche early primates found themselves in in the aftermath of the Permian-Triassic extinction event.

Early primates seemed to have had a unique diet of foraging fruit directly in treetops—they were frugivores. They plucked fruit from trees right after it ripened but before it fell to the forest floor. This allowed primates to have easy access to food without much competition from other species. This unique ecological niche may have offered early primates two gifts that opened the door to their uniquely large brains and complex social groups. First, easy access to fruit gave early primates an abundance of calories, providing the evolutionary option to spend energy on bigger brains. And second, and perhaps more important, it gave early primates an abundance of time.

Free time is extremely rare in the animal kingdom; most animals have no choice but to fill every moment of their daily calendar with

eating, resting, and mating. But these frugivorous primates didn’t have to spend nearly as much time foraging as other animals did, so when seeking to climb the social hierarchy, these primates had a new evolutionary option: instead of spending energy evolving bigger muscles to fight their way to the top, they could spend energy evolving bigger brains to politick their way to the top.

than most other mammals. And it has been shown that this social time is causally related to how much free time primates have; as more free time is given (by providing easier access to food), primates

spend more time socializing.

This created a completely new evolutionary arms race: a battle for political savvy. Any primate born with better tricks for currying favor and gaining allies would survive better and have more babies. This put more pressure on other primates to evolve smarter mechanisms for politicking. Indeed, neocortex size of primates is correlated not only with social-group size but also

with social savviness. An outgrowth of this arms race seems to have been a blossoming of many human social instincts, both the good (friendships, reciprocity, reconciliation, trust, sharing) and the bad (tribalism, nepotism, deception). While many aspects of these behavioral changes did not require any particularly clever new brain systems, there was indeed an intellectual feat underlying this politicking: the ability to engage in theory of mind.

It isn’t clear how political savviness would even be possible if a species did not have at least a basic and primitive version of theory of mind—only through this ability can individuals infer what others want and thereby figure out whom to cozy up to and how. Only through theory of mind can individual primates know not to mess with a low-ranking individual with high-ranking friends; this requires understanding the intent of the high-ranking individuals and what they will do in future situations. Only through this ability of theory of mind can you figure out who is likely to become powerful in the future, whom you need to make friends with, and whom you can deceive.

So this may be why primates began growing such big brains, why their brain size is correlated with social group size, and why primates evolved the ability to reason about the minds of others. The question is, of course, how do primate brains do this?

about three hundred fifty grams. This is almost a thousandfold increase in brain size. Such a large expansion presents a challenge in relating brain areas across time (early mammals to early primates) and across species at a point in time (today’s mouse brain to today’s chimpanzee brain). Which brain areas are truly new and which are merely expanded versions of the same thing?

Clearly some structures in brains will scale naturally with body size without any meaningful change to their function. For example, a bigger body means more touch and pain nerves, which means more neocortical space for processing these sensory signals. The surface area of the somatosensory cortex of the early apes was obviously much bigger than that of early mammals even though it performed the same function. Same for bigger eyes and muscles and anything else that requires incoming or outgoing nerves.

Further, more neurons can be added to a structure to improve its performance without fundamentally changing its function. For example, if the basal ganglia were one hundred times larger, it might enable associations between many more actions and rewards while still fundamentally performing the same function: implementing a temporal difference learning algorithm. Similarly, the visual cortex of primates is massively larger than that of rodents, even accounting for brain scaling. Unsurprisingly, primates are better than rodents at many aspects of visual processing. But the visual area of the neocortex does not perform some unique function in primates; primates simply dedicated more space, proportionally, to the same function and got better performance.

And then there are the fuzzy areas—those structures that are very similar but slightly modified, teetering on the edge of new and old. An example of this is new hierarchical layers of sensory processing in the neocortex. Primates have many hierarchical regions of the visual neocortex with processing hopping from one region to the next. These areas still process visual input, but the addition of new hierarchical layers makes them qualitatively different. Some areas respond to simple shapes; other areas respond to faces.

But there are also, of course, truly new brain regions—structures with completely unique connectivity that perform novel functions.

So the question is: In the brain of early primates, how much was simply scaled up (whether proportionately or disproportionately) and how much was new? Most evidence suggests that despite the dramatic expansion in size, the brain of our primate ancestor, and of primates today, was largely the same as that of the early mammals. A bigger hindbrain, a bigger basal ganglia, a bigger neocortex, but still all the same regions connected in all

the same fundamental ways. These early primates indeed dedicated disproportionately more neocortex to certain functions, such as vision and touch, but still, the functions and connectivity were largely the same. New hierarchical layers were added, with sensory information hopping from one neocortical layer to another, enabling progressively more abstract representations to be formed. But this was mostly just performance-enhancing.

So might it be the case that the surprising smarts of primates, with all their theory of mind, politicking, and trickery, was a consequence of nothing more than brain scaling?

The New Neocortical Regions of Early Primates

Although most of the brain of the early primates was merely a scaled-up mammal brain, there were, in fact, certain areas of neocortex that were truly new. We can categorize these new neocortical areas that emerged within the primate lineage into two groups. The first is the granular prefrontal cortex (gPFC), which was a new

addition to the frontal cortex.

and sensory neocortical regions.

What makes these areas “new”? It isn’t their microcircuitry; all these areas are still neocortex and have the same general columnar microcircuitry as other areas of neocortex across mammals. It is their input and output connectivity that renders them new; it is what these areas construct a generative model of that unlocked fundamentally new cognitive abilities.

As we saw in

functional significance at all. A famous case study at the time was of a seizure patient named K.M. who had a third of his frontal cortex removed to treat seizures. After the surgery, K.M. seemed to have no deficits in

intellect or perception whatsoever. K.M.’s IQ after the removal of a third of his frontal cortex was unchanged—if anything, it increased. In the words of a neuroscientist at the time, the function of granular prefrontal

cortex was a “riddle.”

Modeling Your Own Mind

Consider a study done in 2001. Human subjects were put in an fMRI machine and shown a series of pictures. With each picture, subjects were asked “How does this make you feel?” or some mundane question about the contents of the image, such as “Was this picture taken indoors or outdoors?” Both tasks activated the aPFC, which makes sense, since both require rendering an inner simulation; either of the world around the picture to decide if it is more likely to be inside or outside, or of your own thoughts and sensations. But it was only when humans were asked how they felt about a picture that the

gPFC lit up.

about yourself in general.

With this clue the granular prefrontal cortex is uniquely activated by self-reference, might we have missed some subtle but crucial impairments of granular prefrontal damage?

In 2015, scientists did the following study. They gave participants a neutral cue word (e.g., bird or restaurant) and asked them to tell the experimenter different narratives of themselves associated with that word. Some of these participants were healthy, some had damage to areas of the granular prefrontal cortex, and some had damage to the hippocampus.

How did people’s narratives differ across these conditions? Humans with damage to the granular prefrontal areas but with the aPFC and hippocampus intact were able to imagine complex scenes, rich with detail, but they were impaired at imagining themselves in such scenes. They sometimes even completely omitted themselves from their narratives. Damage to the hippocampus seemed to have the opposite effect—patients could imagine themselves in past or future situations just fine but struggled to build external features of the world; they were unable to describe in detail any

of the surrounding elements.

This suggests that the granular prefrontal cortex plays a key role in your ability to project yourself—your intentions, feelings, thoughts, personality, and knowledge—into your rendered simulations, whether they are about the past or some imagined future. The simulations run in rat brains, which has no gPFC and only an aPFC and hippocampus, show evidence of rendering an external world, but there is nothing to suggest that they truly project any meaningful model of themselves into these simulations.

Damage to the granular prefrontal cortex may also affect people’s models of themselves not only in their mental simulations, but also in the present. Some people with gPFC damage develop mirror-sign syndrome, a condition in which they no longer recognize

themselves in a mirror. These patients insist that the people they see in a mirror are not them. Your modeling of your own mind in the present and the mind you project in your imagination seem to be intimately related.

The idea that the new primate areas take part in modeling your own mind makes sense when you follow their input/output connectivity. The older mammalian aPFC gets input directly from

the amygdala and hippocampus, while the new primate gPFC receives almost no amygdala or hippocampal input or any direct sensory input at all. Instead, the primate gPFC gets most of its input directly

from the older aPFC.

One interpretation of this is that these new primate areas are constructing a generative model of the older mammalian aPFC and sensory cortex itself. Just as aPFC constructs explanations of amygdala and hippocampus activity (invents “intent”), perhaps the gPFC constructs explanations of the aPFC’s model of intent—possibly inventing what one might call a mind. Perhaps the gPFC and PSC construct a model of one’s own inner simulation to explain one’s intentions in the aPFC given knowledge in the sensory neocortex.

Let’s use a thought experiment to build some intuition about what this means. Suppose you put our ancestral primate in a maze. When it reached a choice point, it turned left. Suppose you could ask its different brain areas why the animal turned left. You would get very different answers at each level of abstraction. Reflexes would say, Because I have an evolutionarily hard-coded rule to turn toward the smell coming from the left. Vertebrate structures would say, Because going left maximizes predicted future reward. Mammalian structures would say, Because left leads to food. But primate structures would say, Because I’m hungry, eating feels good when I am hungry, and to the best of my knowledge, going left leads to food. In other words, the gPFC constructs explanations of the simulation itself, of what the animal wants and knows and thinks. Psychologists and philosophers call this metacognition: the ability to think about thinking.

What mammals find in their inner simulations of the external world is, in some sense, the same thing as their knowledge about the external world. When a mammal simulates going down a path and its sensory neocortex renders a simulation that contains water at the end of the path, this is the same thing as “knowing” that water exists at the end of the path. While the older mammalian areas of the sensory neocortex render the simulation of the external world (containing knowledge), the new primate areas of neocortex (what I have been calling the primate sensory cortex) seems to create a model of this knowledge itself (areas of PSC get input from various areas of sensory neocortex). These new primate areas try to explain why the sensory neocortex believes food is over there, why an animal’s inner simulation of the external world is the way it is. An answer might be: Because last time I went over there I saw water, and hence when I simulate going back there, I see water in my imagination. Put slightly differently: Because I last saw water there, I now know that water is over there even though before I did not.

These systems are all bootstrapped on one another. Reflexes drive valence responses without any learning required, making choices based on evolutionarily hard-coded rules. The vertebrate basal ganglia and amygdala can then learn new behaviors based on what has historically been reinforced by these reflexes, making choices based on maximizing reward. The mammalian aPFC can then learn a generative model of this model-free behavior and construct explanations, making choices based on imagined goals (e.g., drinking water). This could be considered a first-order model. The primate gPFC can then learn a more abstract generative model (a second-order model) of this aPFC-driven behavior and construct explanations of intent itself, making choices based on mind states and knowledge (I’m thirsty; drinking water when thirsty feels good, and when I simulate going down this way, I find water in my simulation, hence I want to go in this direction).

The mammalian first-order model has a clear evolutionary benefit: It enables the animal to vicariously play out choices before acting. But what is the evolutionary benefit of going through the trouble of developing a second-order model? Why model your own intent and knowledge?

Modeling Other Minds

Consider the comic-strip task devised by Eric Brunet-Gouet in 2000. Human participants were shown several comic strips, each containing three frames, and asked to guess which fourth-frame ending was most likely. There were two types of comic strips—one required inferring the intent of characters in order to correctly guess the ending, and the other required only understanding physical causal relationships.

Brunet-Gouet performed a PET scan of the subjects’ brains while they did this comic-strip task. He found an interesting difference in which brain regions got excited by each type of comic strip. When asked about comic strips that required understanding the intentions of the characters, but not when asked about other types of comic strips, these uniquely primate areas of neocortex, like gPFC,

lit up with activation.

In addition to inferring other people’s intent, these primate areas also get activated by tasks that require inferring other people’s knowledge. A famous test of this is the Sally-Ann test: Participants are shown a series of events occurring between two individuals, Sally and Ann. Sally puts a marble in a basket; Sally leaves; when Sally isn’t looking, Ann moves the marble from the basket to a nearby box; Sally comes back. The participant is then asked: If Sally wants to play with her marble, where will she look for it?

Answering this correctly requires realizing that Sally has different knowledge than you. While you saw Ann put the marble in the box, Sally did not. And so the right answer is that Sally will look in the basket, even though the marble is not there. There are many forms of the Sally-Ann test, generally referred to as “false-belief tests.” Humans pass such false-belief tests by

the age of four. When humans in fMRI machines are given false-belief tests, the primate gPFC and PSC light up, and people’s performance correlates with

the degree of activation. Indeed, across countless studies, it has been shown that many areas of these uniquely primate neocortical areas are specifically activated by

such false-belief tasks.

If we revisit our mysterious patients with granular prefrontal damage and test them for theory-of-mind tasks, we begin to see a common theme emerge from their seemingly disparate, subtle, and bizarre symptoms. Such patients are worse at solving false-belief tests like the Sally-Ann test; they are much worse at recognizing

emotions in other people; they struggle to empathize

with other people’s emotions, struggle to

distinguish lies from jokes, struggle to identify a faux pas

that would offend someone, struggle to take

someone else’s visual perspective, and

struggle to deceive others.

Although all of the above studies have to do with human brains (albeit the parts of human brains inherited from our primate ancestors), experiments have confirmed the same

effects in nonhuman primates. Put a monkey in a situation where they have to reason about the intent or knowledge of another to solve a task, and gPFC lights up, just as

it does in humans. Damage the gPFC in a monkey, and their performance on such tasks becomes impaired, just as it does in humans.

And indeed, revealing their importance in understanding others, the size of these granular prefrontal areas is correlated with

social-network size in primates. The bigger a primate’s granular prefrontal area, the higher in the social hierarchy it tends to be. This same relationship is even seen in humans: the thicker a human’s granular prefrontal areas, the larger his or her social network, and the better that person’s performance on

theory-of-mind tasks.

These newly primate neocortical regions seem to be the locus of both one’s model of one’s own mind and the ability

to model other minds. The fact that these two seemingly distinct functions have extremely overlapping, if not identical, neural substrates offers a glaring clue as to the evolutionary purpose and mechanism of these new primate structures.

Modeling Your Mind to Model Other Minds

As far back as Plato, there has been a running hypothesis about how humans understand the minds of other humans. The theory is that we first understand our own minds and then use this understanding of ourselves to understand others. Modern formulations of this old idea are referred to as “simulation theory”

or “social projection theory.” When we try to understand why someone else did something, we do so by imagining ourselves in their situation—with their knowledge and life history: “She probably yelled at me because she is stressed out about having a test tomorrow; I know I yell more when I am stressed.” When we try to understand what others will do, we imagine what we would do in their situation if we had their knowledge and their background: “I don’t think James will share his food with George anymore; I believe James saw George steal, and I know if I saw my friend steal from me, I wouldn’t share with him anymore.” We understand others by imagining ourselves in their shoes.

The best evidence for social projection theory is the fact that tasks that require understanding yourself and tasks that require understanding others both activate and require the same uniquely primate neural structures. Reasoning about your own mind and reasoning about other minds is, in the brain,

the same process.

Evidence for social projection theory can also be found in how children develop their sense of self. The childhood development of a sense of self is highly related to the childhood development

of theory of mind. One way to test a child’s sense of self is the mirror self-recognition test. Put a smudge of something on a child’s face, let him look in the mirror, and see if he touches that part of his face, realizing that he is seeing himself. Children don’t tend to pass this test until they are

about two years old. It is also around this time that children begin to exhibit a primitive understanding of mental states and begin to use words like

want, wish, and pretend. It is later, around the age of three, when children realize they can hold their own false beliefs; they are able to make statements such as “I thought it was an alligator. Now I

know it’s a crocodile.” It isn’t until after this point, around the age of four or five, when children pass false-belief tests, such as the Sally-Ann test,

with respect to other people.

Other studies have found a strong correlation between children’s ability to report on their own mental states and their ability to report on others’ mental states—when children get better at one, they tend to simultaneously get

better at the other. Further, damage to the development of one impairs skills in the other; chimpanzees raised in social isolation are unable to recognize

themselves in a mirror. Modeling your own mind and that of others is interwoven.

Our understanding of ourselves often gets cross-wired with our understanding of others, consistent with the idea that we are repurposing a common system for each. For example, being in a specific mood (such as happy or sad) makes you biased to incorrectly infer

these states in others; being thirsty makes you biased to incorrectly think that others are

thirstier than they are. People tend to project their own

personality traits onto others. The self-other distinction can get cross-wired, an effect you would expect if we understand others by projecting ourselves into their situation.

To our original question: How might theory of mind work? One possibility, conceptually at least, might be that the uniquely primate neocortical areas first build a generative model of your own inner simulation (in other words, of your mind) and then use this model to try to simulate

the minds of others.

We are in abstract land here—this is hardly a detailed algorithmic blueprint for how to build an AI system with theory of mind. But the idea of bootstrapping theory of mind by first modeling one’s own inner simulation, of modeling yourself to model others, provides an interesting waypoint. Today, we can train an AI system to watch videos of human behavior and predict what humans will do next; we show these systems endless videos of humans doing things and tell them the right answers of what the humans are doing (“This is handshaking”;

“This is jumping”). Advertising platforms can use behavior to predict what people will buy next. And we do have AI systems that attempt to identify emotions in people’s faces (after the systems have been trained on a bunch of pictures of

faces classified by emotions). But all this is clearly a far cry from the complexity of theory of mind in human (and other primate) brains. If we want AI systems and robots that can live alongside us, understand the type of people we are, deduce what we don’t know that we want to know, infer what we intend by what we say, anticipate what we need or want before we tell them, navigate social relationships with groups of humans, with all of their hidden rules and etiquettes—in other words, if we want true humanlike AI systems, theory of mind will undeniably be an essential component of that system.

In fact, theory of mind might be the most essential aspect of building a successful future with superintelligent AI systems. If superintelligent AI systems cannot infer what we actually mean by what we say, we risk entering a dystopian future where AI systems misinterpret our requests with possibly catastrophic consequences. We will learn more about the importance of theory of mind when making requests of AI systems in

to break open nuts, grass to floss, moss for sponges, clubs to smash beehives, and even twigs

to clean their ears.

In the years since Goodall studied these chimps, tool use has been found all over the animal kingdom. Elephants pick up branches with their trunks to swat

flies and scratch themselves. Mongooses use anvils to break open nuts. Crows use sticks to spear larvae. Octopuses gather large shells to make shields. Wrasses fish have been found to use rocks to break open clams to get

to the inner food.

But primate tool use is more sophisticated than tool use in other animals. Wrasses fish, mongooses, and sea otters have been shown to use tools, but they typically have only a single trick up their sleeve. In contrast, groups of chimpanzees often exhibit over twenty

different tool-using behaviors. Further, with the possible exception of birds and elephants, only primates have been shown to actively manufacture their tools. A chimp will shorten, sharpen, and remove leaves from a stick before using it to fish for termites.

Tool use in primates also shows a remarkable level of diversity across different social groups. Different species of wrasses fish use rocks in the same way despite having no contact with each other. But this is not the case with primates; different groups of the same species of primate exhibit surprisingly unique tool-using behaviors. The chimpanzees of Goualougo manufacture termite fishing rods differently

than those in Gombe. Some groups of chimps regularly use rocks to open nuts; other groups don’t. Some groups of chimps use clubs to pound beehives; other groups don’t. Some groups use leafy twigs to swat away flies; others don’t.

If the driver of brain evolution in early primates was a politicking arms race, why would primates be uniquely good tool users? If the new brain regions of primates were “designed” to enable theory of mind, then from where do the unique tool-using skills of primates emerge?

Monkey Mirrors

In 1990, Giuseppe di Pellegrino, Leonardo Fogassi, Bittorio Gallese, and Luciano Fadiga were chowing down lunch in the lab. They were members of Giacomo Rizzolatti’s neurophysiology lab at the University of Parma; their mandate was to study the neural mechanisms of primate fine motor skills. A few feet from the lunch table sat a macaque monkey who was the object of their study. They had placed electrodes throughout its brain to search for which areas responded to the execution of specific types of hand movements. They had found specific areas of premotor cortex that activated when monkeys performed specific types of hand movements, some areas for grasping, others for holding,

and others for tearing. But through lucky happenstance, they were about to discover something much more remarkable. As one of the lab members picked up a sandwich and bit into it, a loud crackle buzzed from a nearby speaker. The noise didn’t come from the speaker connected to the fire alarm or the record player, it came from the speaker connected to the monkey’s brain.

As they now recall, they had an immediate perception that something important

had just happened. They had attached electrodes only to the motor areas of neocortex, the areas that were supposed to activate only when the monkey itself performed specific hand movements. But at that moment during lunch, despite the monkey not moving at all, the same area for hand grasping lit up at the exact moment when one of the human lab members grasped their own food.

After trying to replicate this phantom sandwich-watching activation, Rizzolati’s team quickly realized that they had, in fact, discovered something more general: when their monkey observed a human perform a motor skill—whether picking up a peanut with two fingers, grasping an apple with their full hand, or grasping a snack with their mouth—the monkey’s own motor neurons for performing that same skill would often activate. In other words, the neurons in the premotor and motor areas of a monkey’s neocortex—those that control a monkey’s own movements—not only activated when they performed those specific fine motor skills, but also when they merely watched others perform them. Rizzolatti called these “mirror neurons.”

Over the subsequent twenty years, Rizzolatti’s mirror neurons have been found in numerous behaviors (grasping, placing, holding, finger movements, chewing, lip smacking,

sticking one’s tongue out), across multiple areas of the brain (premotor cortex,

parietal lobe, motor cortex), and across

numerous species of primates. When a primate watches another primate do an action, its premotor cortex often mirrors the actions it is observing.

There are multiple competing

interpretations of mirror neurons. Some argue mirror neurons are nothing more than associations—motor neurons get activated in response to any cue that has been associated with a movement. Monkeys see their own arms make grasping movements when they choose to make grasping movements, so of course when they see someone else’s arms grasp things, some of the same neurons get a small kick of excitement. Others argue that mirror neurons represent something more fundamental—perhaps mirror neurons are the mechanism by which primates engage

in theory of mind. The hypothesis is that primates have some clever mechanism for automatically mentally mirroring movements they see in others, and then by modeling themselves doing that behavior, they can ask “Why would I do this?” and try to deduce another monkey’s or human’s intentions.

Others have a more middle-of-the-road interpretation. Perhaps mirror neurons don’t have some automatic mechanisms of mirroring; they are just clues that monkeys happen to be imagining themselves doing what they

see someone else do. Mirror neurons aren’t anything special; they are just evidence that monkeys are thinking about themselves grasping food when they see you grasp food. And as we already saw in

open (without seeing anything). Similarly, the neurons in a monkey that activate when it picks up a box will also activate when the monkey sees a human presumably pick up a box that is obscured behind a wall (but the neurons don’t activate if monkeys know there is no

box behind the wall). If mirror neurons were simply automatic mirrors, then they wouldn’t activate in the above cases where monkeys were not directly observing behaviors. However, if mirror neurons are the consequence of imagined behaviors, then whenever something triggers a monkey to imagine itself doing something, you would see these mirror neurons activate.

If we accept the interpretation of mirror neurons as the imagination of movements, then this begs the question why monkeys tend to imagine themselves doing what they see others doing. What is the point of mentally simulating movements you see in others? In

phrase, like combing hair). Individuals who struggle to brush their own teeth tend to be bad at recognizing teeth-brushing in others.

Further, temporarily inhibiting a human’s premotor cortex impairs their ability to correctly infer the weight of a box when watching a video of someone picking it up (arms that easily pick it up suggest it is light, but arms that struggle at first and have to adjust their position to get more leverage suggest it is heavy), but it has no impact on their ability to infer a ball’s weight by watching a video of it

bouncing on its own. This suggests that people mentally simulate themselves picking up a box when seeing someone else pick up a box (“I would turn my arm that way only if the box was heavy”).

This impairment in understanding the actions of others is not some generalized effect of premotor cortex interference but is highly specific to the body parts that a brain is prevented from simulating. For example, temporarily inhibiting the hand area of your premotor cortex (which simulates your own hand movements), not only impairs your ability to perform your own hand movements, but also impairs your ability to recognize mimed hand movements (such as correctly recognizing mimed movements of grasping a hammer or pouring tea), but it has no impact on your ability to recognize mimed mouth movements (such as identifiying licking ice cream,

eating a burger, blowing out a candle). Conversely, temporarily inhibiting the mouth area of your premotor cortex does impair your ability to recognized mimed mouth movements, while having no impact on your ability to recognize hand movements.

This suggests that the premotor cortex and motor cortex, the brain regions required to simulate your own movements, are also required to simulate the movements of others to understand their actions. But by understand, we don’t mean understanding others’ emotions (hungry, fearful) or others’ knowledge (“Does Bill know that Jane hid the food?”). These studies demonstrate that the premotor cortex is involved in specifically understanding the sensorimotor aspects of others’ behavior—inferring the strength required to pick up a box or the type of tool someone

intended to hold.

But why does it matter to correctly identify the sensorimotor aspects of the behaviors you observe in others? What benefit does it provide to realize the tool someone is trying to hold or the weight of a box? The main benefit is that it helps us, as it helped early primates, learn new skills through observation. We already saw in

becomes way more activated than when they observe a chord they already know how to play.

But the fact that premotor cortex becomes uniquely activated when trying to learn a new skill through observation, does not prove that it is required to learn a new skill through observation. Suppose you ask a human to watch two different videos. In the first video, he sees a hand pushing specific buttons on a keyboard, and he is asked to imitate those hand movements and push his own version of the same keyboard. In the other video, he sees a red dot move to different buttons on a keyboard, and he is asked to push the same buttons on his own keyboard. If you temporarily inhibit his premotor cortex during this task, he becomes specifically impaired at imitating hand motions but performs normally at

following the red dots. Premotor activation is not just correlated with imitation learning; it seems to be, at least in some contexts, necessary for imitation learning. And here we can begin to unravel why primates are such great tool users.

Transmissibility Beats Ingenuity

Think about all the clever motor skills related to using tools: typing, driving, brushing your teeth, tying a tie, or riding a bicycle. How many of these skills did you figure out on your own? I’m going to bet that practically all these skills were acquired by observing others, not by your own independent ingenuity. Tool use in nonhuman primates originates the same way.

Most chimps in a group use the same tool techniques not because they all independently came up with the same trick but because they learned by observing each other. The amount of time a young chimp spends watching its mother using termite fishing tools or ant dipping tools is a significant predictor of the age at which it will learn each skill; the more it watches, the earlier it

learns to do it. Without transmission from others, most chimps never figure tool use out on their own; in fact, a young chimp that doesn’t learn, through observing others, to crack nuts by the age of five will not acquire the

skill later in life.

food in the cage).

These skills can propagate throughout an entire group of primates. Consider the following studies. Experimenters temporarily took an individual chimpanzee, capuchin monkey, or marmoset from their group and taught it a new skill. These individuals were taught skills such as using a stick to poke a food-dispensing device

in the right way, sliding open a door in a specific

way to get food, pulling a

drawer to get food, or

opening an artificial fruit. After teaching this new skill, experimenters reintroduced the now skilled primate back to their group. Within a month, almost the entire group was using these same techniques, whereas groups that never had a member taught the skill never figured out how to use the tools in the same way. And such skills, originally taught to only a single individual, were passed

down through multiple generations.

The ability to use tools is less about ingenuity and more about transmissibility. Ingenuity must occur only once if transmissibility occurs frequently; if at least one member of a group figures out how to manufacture and use a termite-catching stick, the entire group can acquire this skill and continuously pass it down throughout generations.

But it would be inaccurate to conclude that primates are uniquely good tool users simply because of their ability to transmit motor behaviors among themselves—to learn by observation. Many animals who are much worse tool users than primates, even those who don’t use tools at all, also engage in observational learning. Rats can learn to push a lever to get water by watching another rat push a

lever and get water. Mongooses will adopt the egg-opening

technique of their parents. Dolphins can be trained to imitate movements they see in

other dolphins or humans. Dogs can learn how to pull a lever with their paw to get food by watching another

dog perform the act. Even fish and reptiles can observe the navigational paths taken by other members of their own species and learn to take

those same navigational paths.

SELECTING KNOWN SKILLS THROUGH OBSERVATION	ACQUIRING NOVEL SKILLS THROUGH OBSERVATION
Many mammals Octopuses Fish Reptiles	Primates Some birds

But there is a difference between observational learning in primates relative to most other mammals. If a parent mongoose tends to break open eggs with its mouth, then so does its offspring; if a parent mongoose tends to break open eggs by throwing, then so does its offspring. But these children mongooses aren’t acquiring a novel skill by watching; they are merely changing which technique they tend to use—all children mongooses exhibit both biting and throwing tricks for opening eggs. Kittens learn to pee in litter boxes only if exposed to their mothers doing it, but all kittens know how to pee. Fish don’t learn how to swim by watching; they merely change their paths by watching. In all these cases, animals aren’t using observational learning to acquire novel skills; they are merely selecting a known behavior based on seeing another do the same thing.

Selecting a known behavior through observation can be accomplished with simple reflexes: A tortoise may have a reflex to look in the direction that other tortoises are looking; a fish may have a reflex to follow other fish. A mouse can simulate itself pushing a lever (something it already knows how to do) when it observes another mouse pushing a lever, at which point this mouse will realize that it will get water if it does this. But acquiring an entirely novel motor skill by observation may have required, or at least hugely benefited from, entirely new machinery.

Why Primates Use Hammers but Rats Don’t

Acquiring novel skills through observation required theory of mind, while selecting known skills through observation did not. There are three reasons why this was the case. The first reason why theory of mind was necessary for acquiring novel skills by observation is that it may have enabled our ancestors to actively teach. For skills to be transmitted through a population, you don’t need teachers—dutiful observation by novices will do. But active teaching can substantially improve the transmission of skills. Think about how much harder it would have been to learn to tie your shoes if you didn’t have a teacher who slowed down and walked you through each step, and instead you had to decipher the steps on your own by watching people rapidly tie their shoes with no regard for your learning.

Teaching is possible only with theory of mind. Teaching requires understanding what another mind does not know and what demonstrations would help manipulate another mind’s knowledge in the correct way. While it is still controversial whether any primates other than humans teach, in recent years evidence is beginning to accumulate in favor of the idea that nonhuman primates do, in fact,

actively teach one another.

In the 1990s, the primatologist Christophe Boesch reportedly observed chimp mothers performing nut cracking in slow motion specifically around their young. These mothers periodically looked over to ensure their child was paying attention. Boesch reported chimp mothers correcting mistakes in youngsters by removing a nut, cleaning the anvil, then placing the nut back. He also reported mothers reorienting a hammer in the

hands of their young.

Monkeys have been found to exaggerate their “flossing” specifically around youngsters that have not yet learned this skill, as if they are slowing

down to help teach. Chimps that are skilled termite fishers will often bring two sticks to a fishing activity and directly hand one to a youngster. A chimp will even break its own stick in two and give the youngster half if the child shows up without a stick. If a child starts to seem like it is struggling with a task, the mother will actively

swap tools with them. And the more complex the tool-using process, the more likely a mother is to give a

tool to a youngster.

The second reason why theory of mind was necessary for learning novel motor skills through observation is that it enabled learners to stay focused on learning over long periods. A rat can see another rat push a lever and a few moments later push the lever itself. But a chimpanzee child will watch its mother use anvils to break open nuts and practice this technique for years without any success before it begins to master the skill. Chimp children continually attempt to learn without any near-term reward.

It is possible chimp children do this simply because they find imitation rewarding on its own, but another possibility is that theory of mind enables novices to identify the intent of a complex skill, which makes them highly motivated to keep trying to adopt it. Theory of mind enables a chimp child to realize that the reason it is not getting food with its stick while its mother is getting food is that its mother has a skill it does not yet have. This enables a continuous motivation to acquire the skill, even if it takes a long time to master. When a rat imitates behaviors, on the other hand, it will quickly give up if its actions don’t lead to a near term reward.

The third and final reason why theory of mind was necessary for learning novel motor skills through observation was that it enabled novices to differentiate between the intentional and unintentional movements of experts. Observational learning is more effective if one is aware of what another is trying to accomplish with each movement. If you watched your mother tie her own shoes and you had no idea what aspects of her movements were intentional versus accidental, it would be quite hard to decipher which movements to follow. If you realized that her intention was to get her shoes tied, that when she slipped it was an accident, and that both the way she is seated and the angle of her head are irrelevant aspects of the skill, it would be much easier for you to learn the skill through observation.

This is indeed how chimpanzees learn. Consider the following experiment. An adult chimpanzee was allowed to observe a human experimenter open a puzzle box to get food. Throughout the sequence of actions needed to open the puzzle box, the experimenter did several irrelevant actions, such as tapping a wand or rotating the box. Then the chimp was given a chance to open the puzzle box to get food itself. Amazingly, the chimp didn’t exactly copy every movement of the experimenter, instead it copied only the necessary movements to open the puzzle box and

skipped the irrelevant steps.

Understanding the intentions of movements is essential for observational learning to work; it enables us to filter out extraneous movements and extract the essence of a skill.

Robot Imitation

In 1990, a graduate student at Carnegie Mellon named Dean Pomerleau and his adviser Chuck Thorpe built an AI system to autonomously drive a car. They called it ALVINN (Autonomous Land Vehicle in a Neural Network). ALVINN was fed video footage from around a car and could—on its own—steer the car to stay within a lane on a real highway. There had been previous attempts at autonomous cars like this, but they were very slow, often pausing every few seconds; the original version of the autonomous car in Thorpe’s group could go only a quarter of a mile per hour due to how much thinking it had to do. ALVINN was much faster, so fast, in fact, that ALVINN successfully drove Pomerleau from Pittsburgh to the Great Lakes on a real highway with other drivers.

Why was ALVINN successful while previous attempts had failed? Unlike previous attempts to build a self-driving car, ALVINN was not taught to recognize objects or plan its future movements or understand its location in space. Instead, ALVINN outperformed these other AI systems by doing something simpler: It learned by imitating human drivers.

Pomerleau trained ALVINN the following way: He put a camera on his car and recorded both the video and the position of his steering wheel while he drove around.

a handful of laps. This is not unlike a teacher chimpanzee correcting the movements of a child learning a new skill. A chimp mother watches her daughter try to insert a stick into a termite mound, and when the child is struggling, the mother attempts to correct her.

The second approach to imitation learning in robotics is

called “inverse reinforcement learning.” Instead of trying to directly copy the driving decisions that a human makes in response to a picture of the road, what if an AI system first attempts to identify the intent of the human’s driving decisions?

In 2010, Pieter Abbeel, Adam Coates, and Andrew Ng demonstrated the power of inverse reinforcement learning by using it to get an AI system to autonomously fly

a remote-controlled helicopter. Flying a helicopter, even a remote-controlled one, is hard; helicopters are unstable (small mistakes can rapidly cascade into a crash), require constant adjustments to keep them in the air, and require correctly balancing multiple complex inputs at the same time (angle of top blades, angle of tail rotor blades, tilt orientation of the helicopter body, and more).

Ng and his team didn’t want to get an AI system to simply fly a helicopter, they wanted it to perform acrobatic tasks, those that only the best human experts could perform: flipping in place without falling, rolling while moving forward, flying upside down, performing arial loops, and more.

Part of their approach was standard imitation learning. Ng and his team recorded human expert inputs to the remote control as they performed these acrobatic tricks. But instead of training the AI system to directly copy the human experts (which didn’t work), they trained the AI system to first infer the intended trajectories of the experts, inferring what it seemed like the humans were trying to do, and then the AI system learned to pursue those intended trajectories. This technique is called “inverse reinforcement learning” because these systems first try to learn the reward function they believe the skilled expert is optimizing for (i.e., their “intent”), and then these systems learn by trial and error, rewarding and punishing themselves using this inferred reward function. An inverse reinforcement learning algorithm starts from an observed behavior and produces its own reward function; whereas in standard reinforcement learning the reward function is hard-coded and not learned. Even when expert pilots flew these helicopters, they were continually recovering from small mistakes. By first trying to identify the intended trajectories and movements, Ng’s AI system was both filtering out extraneous mistakes of experts, and correcting its own mistakes. Using inverse reinforcement learning, by 2010 they successfully trained an AI system to autonomously perform arial aerobatics with a helicopter.

There is still much work to be done when it comes to

imitation learning in robotics. But the fact that inverse reinforcement learning (whereby AI systems infer the intent of observed behavior) seems necessary for observational learning to work, at least in some tasks, supports the idea that theory of mind (whereby primates infer the intent of observed behavior) was required for observational learning and the transmission of tool skills among themselves. It is unlikely a coincidence that both the ingenuity of roboticists and the iteration of evolution both converged on similar solutions; a novice cannot reliably acquire a new motor skill by merely observing an expert’s movements, novices must also peer into an expert’s mind.

Theory of mind evolved in early primates for politicking. But this ability was repurposed for imitation learning. The ability to infer the intent of others enabled early primates to filter out extraneous behaviors and focus only on the relevant ones (what did the person mean to do?); it helped youngsters stay focused on learning over long stretches of time; and it may have enabled early primates to actively teach each other by inferring what a novice does and does not understand. While our ancestral mammal likely could select known skills by observing others, it was with early primates, armed with theory of mind, when the ability to acquire truly novel skills through observation emerged. This created a new degree of transmissibility: skills that were discovered by clever individuals and that would once have faded when they died, could now propagate throughout a group and be passed down endlessly through generations. This is why primates use hammers and rats do not.

less than seventy-two hours. Some trees offer ripe fruit for less than three weeks of the year. Some fruit has few animal competitors (such as bananas in their hard-to-open skin), while other fruit has many animal competitors (such as figs, which are easy for any animal to eat). These popular fruits are likely to disappear quickly, as many different animals feed on them once they ripen. Altogether this meant that primates needed to keep track of all the fruit in a large area of forest and on any given day know which fruit was likely to be ripe; and of the fruit that was ripe, which was likely to be most popular and hence disappear first.

Studies have shown that chimpanzees plan their nighttime nesting locations in preparation for foraging on the subsequent day. For fruits that are more popular, such as figs, they will go out of their way to plan where they sleep to be en route to these fruits. They do not do the same for less competitive but equally enjoyable fruits. Further, chimpanzees will leave earlier in the morning when pursuing a competitive fruit than when traveling to forage for

a less competitive fruit. It has been shown that baboons also plan their foraging journey in advance and will leave earlier when fruit is less abundant and likely

to be depleted more quickly.

Animals who feed on non-fruit plants do not have to cope with this same challenge; leaves, nectar, seeds, grass, and wood all last for long periods and are not localized in sparse patches. Even carnivores don’t have as cognitively challenging a task—prey must be hunted and outsmarted, but there are rarely only short time windows in which hunting is possible.

Part of what makes this frugivore strategy so challenging is that it requires not only simulating differing navigational paths but also simulating your own future needs. Both a carnivore and a non-fruit-eating herbivore can survive by hunting or grazing only when they are hungry. But a frugivore must plan its trips in advance before it is hungry. Setting up camp en route to a nearby popular fruit patch the night before requires anticipating the fact that you will be hungry tomorrow if you don’t take preemptive steps tonight to get to the food early.

Other mammals, such as mice, clearly stock up on food as winter months approach, storing vast reserves of nuts in their burrows to survive the long stretch when trees produce little to no food. But such seasonal hoarding is not nearly as cognitively challenging as the daily need to change your plans based on how hungry you will be tomorrow. Further, it isn’t even clear that mice hoard food because they understand that they will be hungry in the future. Indeed, lab mice—although they have never suffered from a cold winter without food—automatically start hoarding food if you simply lower the temperature of their environment, an effect seen only in northern species of mice who have had to

evolve to survive winters. Therefore, this doesn’t seem to be an activity that they have learned from past winters and cleverly responded to; it seems that such hoarding is an evolutionarily hard-coded response to the changing seasons.

The ecological-brain hypothesis argues that it was the frugivore diet of early primates that drove the rapid expansion of their brains. In 2017, Alex DeCasien from NYU published a study examining the diets and social lives of over

140 species of primates. Some primates are primarily frugivores; others are now primarily folivores (feeding on leaves). Some primates live in very small social groups; others live in large ones. He surprisingly found that being a frugivore seemed to explain the variation in relative brain size perhaps even better than the size of a primate’s social group.

In the 1970s, two comparative psychologists by the name of Doris Bischof-Kohler and her husband, Norbert Bischof, proposed a novel hypothesis about what was unique about planning in humans: They hypothesized that while other animals can make plans based on current needs (like how to get to food when they are hungry), only humans can make plans based on future needs (like how to get food for your trip next week, even though you are not hungry right now). The evolutionary psychologist Thomas Suddendorf would later call this the “Bischof-Kohler hypothesis.”

Humans anticipate future needs all the time. We go grocery shopping even when we are not hungry; we bring warm clothes on trips even when we are not cold. Given the evidence available at the time of Bischof-Kohler, it was a reasonable hypothesis that only humans could do this. But recent evidence has called this into question. There are now anecdotal stories of chimpanzees bringing straw from inside a warm cage to make a nest outside when they know it is cold outside but

before they were cold. There have been findings of bonobos and orangutans selecting tools for future tasks up to fourteen hours in advance

of that task. Chimpanzees will carry stones from faraway locations to open nuts in areas that

have no suitable stones and will manufacture tools in one location for

use in another location. Indeed, if frugivorism requires planning ahead before you are hungry, then we should expect primates to be able to anticipate future needs.

In 2006, Miriam Naqshbandi and William Roberts of the University of Western Ontario measured a squirrel monkey’s and a rat’s ability to anticipate their own future thirst and

change their behavior accordingly. Squirrel monkeys and rats were both given two options by being presented with two cups. Cup 1 was a “small treat option,” which contained a tiny morsel of food, and cup 2 was a “high treat option,” which contained lots of food. For squirrel monkeys the treats were dates; for rats they were raisins. Under normal conditions, both animals would choose the high treat option; they love dates and raisins.

But Naqshbandi and Roberts then tested these animals in a different condition. Dates and raisins induce large amounts of thirst in these animals, often requiring them to consume over twice as much water to rehydrate themselves. So what happens if these animals are forced to make a trade-off, incorporating their future state of thirst? Naqshbani and Roberts modified the test such that if animals select the high treat option (the cup with many dates or raisins), they will only get access to water hours later; however, if animals select the low treat option (the cup with few dates or raisins) they get access to water between 15 and 30 minutes later. What happens?

Fascinatingly, squirrel monkeys learn to select the low treat option, while rats continue to select the high treat option. Squirrel monkeys are capable of resisting the temptation to have treats now, in anticipation of something—water—that they don’t even want yet. In other words, monkeys can make a decision in anticipation of a future need. In contrast, rats were entirely unable to do this—they stuck with the flawed logic of “why give up extra raisins for water, I’m not even thirsty!”

hunger will be satisfied.

Although Naqshbandi and Roberts’s experiment with squirrel monkeys and rats suggests that Suddendorf may have been wrong that only humans can anticipate future needs, Suddendorf may have been prescient in proposing that the general ability to model a dissociated mental state from your own can be repurposed for both theory of mind

and anticipating future needs.

There are two observations that support this idea. First, it seems that both theory of mind and anticipating future needs are present, even in a primitive form, in primates, but not in many other mammals, suggesting both abilities emerged around the same time in early primates. Second, people make similar types of mistakes in tasks of theory of mind and of anticipating future needs.

For example, we saw in

that were well fed, even though they were both shopping for food to feed themselves for the same window of time, namely, a single week. When hungry, you overestimate your own future hunger.

The ability to anticipate future needs would have offered numerous benefits to our ancestral frugivores. It would have enabled our ancestors to plan their foraging routes long in advance, thereby ensuring they were the first to get newly ripened fruits. Our ability to make decisions today for faraway, abstract, and not-yet-existent goals was inherited from tree faring primates. A trick that, perhaps, was first used for getting the first pick of fruits, but today, in humans, is used for far greater purposes. It laid the foundation for our ability to make long term plans over vast stretches of time.

at similar developmental times. We see this in the fact that damage that impairs one of these abilities tends to impair many of them.

And most important, we see this in the fact that the structures from which these skills emerge are the same areas from which our ability to reason about our own mind emerges. These new primate areas are required not only for simulating the mind of others but also for projecting yourself into your imagined futures, identifying yourself in the mirror (mirror-sign syndrome), and identifying your own movements (alien-hand syndrome). And a child’s ability to reason about her own mind tends to precede a child’s development of all three of these abilities.

However, the best evidence for this idea goes all the way back to Mountcastle. The main change to the brain of early primates, besides its size, was the addition of new areas of neocortex. So if we are to stick to the general idea—inspired by Mountcastle, Helmholtz, Hinton, Hawkins, Friston, and many others—that every area of neocortex is made up of identical microcircuits, this imposes strict constraints on how we explain the newfound abilities of primates. It suggests that these new intellectual skills must emerge from some new clever application of the neocortex and not some novel computational trick. This makes the interpretation of theory of mind, imitation learning, and anticipation of future needs as nothing more than an emergent property of a second-order generative model a nice proposal—all three abilities can emerge from nothing more than new applications of neocortex.

All these abilities—theory of mind, imitation learning, and anticipating future needs—would have been particularly adaptive in the unique niche of early primates. Dunbar argues that the social-brain hypothesis and the ecological-brain hypothesis are two sides of the same coin. The ability to mentalize may have simultaneously unlocked both the ability to successfully forage fruits and to successfully politick. The pressures of both frugivorism and social hierarchies may have converged to produce continual evolutionary pressure to develop and elaborate brain regions—such as the gPFC—for modeling your own mind.

We have reached the end of

ago in eastern Africa. The offspring of this ancestor split down two evolutionary paths—one that became today’s chimpanzees, and another that became today’s humans.

If we were to scrunch the six hundred million years of evolutionary time—from which the first brains emerged until today—into a single calendar year, then we would now find ourselves perched at Christmas Eve, the final seven days of December. Over the next “seven days,” our ancestors will go from foraging fruits to flying Falcon 9 rockets. Let’s find out how.

and not of kind.” What intellectual feats, if any, are uniquely human is still hotly debated among psychologists. But as the evidence continues to roll in, it seems that Darwin may have been right.

If it were the case that humans wielded numerous intellectual capabilities that were entirely unique in kind, we would expect human brains to contain some unique neurological structures, some new wiring, some new systems. But the evidence is the opposite—there is no neurological structure found in the human brain that is not also found in the brain of our fellow apes, and evidence suggests that the human brain is literally just a scaled-up primate brain: a bigger neocortex, a bigger basal ganglia, but still containing all the same areas wired

in all the same ways. Scaling up a chimpanzee brain may have made us better at anticipating future needs, theory of mind, motor skills, and planning, but it didn’t necessarily give us anything truly new.

An eminently reasonable explanation of human brain evolution since our divergence with chimpanzees is that various evolutionary pressures led our human ancestors to simply “level up” the abilities that were already present.

Was there perhaps no breakthrough at all?

This seems to be the most reasonable interpretation—but for one crucial exception. And it is in this singular exception that we see the first hints of what it means to be human.

Our Unique Communication

Organisms had been communicating with each other long before early humans uttered their first words. Single-celled organisms emit chemical signals to share genes and information about the environment. Brainless sea anemones pump pheromones into the water to coordinate the timing of sperm and egg release. Bees dance to signal where to find food. Fish use electrical signals to court each other. Reptiles head-bob to communicate their aggressiveness. Mice squeak to express danger or excitement. Communication between organisms is evolutionarily ancient and ubiquitous.

Chimpanzees are constantly screeching and gesturing to each other. These different

sounds and gestures have been shown to signal specific requests. Shoulder taps mean “Stop it,” stomping means “Play with me,” a screech means “Groom me,” reaching a palm out means “Share food.” Primatologists have studied these different gestures and vocalizations so extensively that there is even a Great Ape Dictionary that has chronicled almost one hundred sounds and gestures.

Vervet monkeys have different sounds to signal the presence of specific predators. When one monkey makes the squeal meaning “Leopard!” all the others run to the trees. When one makes the squeal meaning “Eagle!” all the others jump to the forest floor. Experimenters can get all the monkeys to run to the treetops or jump to the floor simply by playing one of these sounds from nearby loudspeakers.

And then, of course, there is us—Homo sapiens. We too communicate with each other. It is not the fact that we communicate that is unique; rather, it is how we communicate. Humans use language.

Human language differs from other forms of animal communication in two ways. First, no other known form of naturally occurring animal communication assigns declarative labels (otherwise known as symbols). A human teacher will point to an object or a behavior and assign it an arbitrary label: elephant, tree, running. In contrast, other animals’ communications are genetically hardwired and not assigned. Vervet monkey and chimpanzee gestures are almost identical across different groups that have no contact with each other. Monkeys and apes deprived of social contact still use the same gestures. In fact, these gestures are even shared across species of primates; bonobos and chimpanzees share almost exactly the same repertoire of gestures and vocalizations. In nonhuman primates, the meanings of these gestures and vocalizations is not assigned through declarative labeling but emerge directly from genetic hardwiring.

What about teaching a dog or any other animal a command? Clearly this represents some form of labeling. Linguists make a distinction between declarative and imperative labels. An imperative label is one that yields a reward: “When I hear sit, if I sit, I will get a treat” or “When I hear stay, if I stop moving, I will get a treat.” This is basic temporal difference learning—all vertebrates can do this. Declarative labeling, on the other hand, is a special feature of human language. A declarative label is one that assigns an object or behavior an arbitrary symbol—“That is a cow,” “That is running,”—without any imperative at all. No other form of naturally occurring animal communication has been found to do this.

The second way in which human language differs from other animal communication is that it contains grammar. Human language contains rules by which we merge and modify symbols to convey specific meanings. We can thereby weave these declarative labels into sentences, and we can knit these sentences into concepts and stories. This allows us to convert the few thousand words present in a typical human language into a seemingly infinite number of unique meanings.

The simplest aspect of grammar is that the order in which we utter symbols conveys meaning: “Ben hugged James” means something different from “James hugged Ben.” We also embed subphrases that are sensitive to ordering: “Ben, who was sad, hugged James” means something entirely different than “Ben hugged James, who was sad.” But the rules of grammar go beyond just order. We have different tenses to convey timing: “Ben is attacking me” versus “Max attacked me.” We have different articles: “The pet barked” means something different than “A pet barked.”

And of course, this is just English; there are over six thousand spoken languages on Earth, each with its own labels and grammar. But despite the great diversity of the specific labels and grammars of different languages, every single group of humans ever discovered used language. Even hunter-gatherer societies in Australia and Africa who at the point they were “discovered” had had no contact with any other groups of humans for over fifty thousand years, still spoke their own languages that were equally complex as those of other humans. This is irrefutable evidence that the shared ancestor of humans spoke their own languages, with their own declarative labels and grammars.

Of course, the fact that early humans spoke in their own languages with declarative labels and grammar, while no other animal naturally does so, does not prove that only humans are able to use language, merely that only humans happen to use language. Did the brains of early humans really evolve some unique ability to speak? Or is language just a cultural trick that was discovered over fifty thousand years ago and was simply passed down through generations of all modern humans? Is language an evolutionary invention or a cultural invention?

Here’s one way to test this: What happens if we try to teach language to our evolutionarily closest animal cousins, our fellow apes? If apes succeed in learning language, that suggests that language was a cultural invention; if apes fail, this suggests their brains lack a key evolutionary innovation that emerged in humans.

This test has been performed multiple times. The result is as surprising as it is revealing.

Attempts to Teach Apes Language

To start: We can’t literally teach apes to speak. This was attempted in the 1930s, and it failed—nonhuman apes are physically incapable of producing verbal language. Human vocal cords are uniquely adapted to speech; the human larynx is lower and the human neck is longer, which enables us to produce a much wider variety of vowels and consonants than other apes. The vocal cords of a chimp can produce only a limited repertoire of huffs and squeals.

However, what makes language language is not the medium but the substance—many forms of human language are nonverbal. No one would claim that writing, sign language, and Braille do not contain the substance of language because they don’t involve vocalization.

The key studies that attempted to teach chimpanzees, gorillas, and bonobos language used either American Sign Language or made-up visual languages in which apes pointed to sequences of symbols on a board. Beginning as infants, these apes were trained to use these languages, with human experimenters signing or pointing to symbols to refer to objects (apples, bananas) or actions (tickling, playing, chasing) over and over again until the apes began to repeat the symbols.

Across most of these studies, after years of being taught, nonhuman apes did indeed produce the appropriate signs. They could look at a dog and sign dog and look at a shoe and sign shoe.

They could even construct basic noun-verb pairs. Common phrases were play me and tickle me. Some evidence even suggested they could combine known words to create novel meanings. In one famous anecdote, the chimpanzee Washoe saw a swan for the first time, and when the trainer signed, What’s that?, Washoe signed back, Water bird. In another, the gorilla Koko saw a ring and, not knowing the word for it,

signed, Finger bracelet. After eating kale for the first time, Kanzi the bonobo pressed symbols for slow lettuce.

Kanzi supposedly even used language to play with others. There is an anecdotal story of a trainer who was resting in Kanzi’s habitat being woken up by Kanzi snatching away the blanket and then excitedly pressing the symbols for bad surprise. In another anecdote, Kanzi pressed keys for apple chase and then picked up an apple, grinned, and began

running from his trainer.

Sue Savage-Rumbaugh, the psychologist and primatologist who devised the Kanzi language-learning experiment, did a test to compare Kanzi’s language understanding to that of a two-year-old human child. Savage-Rumbaugh exposed Kanzi and a human child to over six hundred novel sentences (using their symbol-language) with specific commands. These sentences used symbols Kanzi already knew but in sentences Kanzi had never seen, commands like Can you give the butter to Rose?; Go put some soap on Liz; Go get the banana that’s in the refrigerator; Can you give the doggy a hug?; and Put on the monster mask and scare Linda. Kanzi successfully completed these tasks over

70 percent of the time, outperforming the two-year-old human child.

The degree to which these ape language studies demonstrate language with declarative labels and grammar is still controversial among linguists, primatologists, and comparative psychologists. There are many who argue that these tricks represent imperatives not declaratives and that the phrases uttered were so simple they could hardly be called grammar. Indeed, in most of these studies, apes received treats when they used the right labels, which makes it hard to tell if they were really sharing in labeling objects or just learning that if they did sign X when they saw a banana, they got a treat, a task any model-free reinforcement learning machine could perform. Extensive analysis of the phrases uttered by these language-able apes demonstrate a low diversity in phrases, meaning they tended to use the exact phrases they learned (e.g., Tickle me) instead of combining words into novel phrases (e.g.,

I want to be tickled). But in response to these challenges, many point to the Savage-Rumbaugh studies and Kanzi’s incredibly accurate grammatical understanding of commands and playful phrases. The debate is not yet settled.

On balance, most scientists seem to conclude that some nonhuman apes are indeed capable of learning at least a rudimentary form of language but that nonhuman apes are much worse than humans at it and don’t learn it without painstaking deliberate training. These apes never surpass the abilities of a young human child.

So, language seems unique to humans on two counts. First, we have a natural tendency to construct it and use it, which other animals do not. Second, we have a capacity for language that far surpasses that of any other animal, even if some basic semblance of symbols and grammar is possible in other apes.

But if language is what separates us from the rest of the animal kingdom, then what is it about this seemingly innocuous trick that enabled Homo sapiens to ascend to the top of food chain; what is it about language that makes those who wield it so powerful?

Transferring Thoughts

Our unique language, with declarative labels and grammar, enables groups of brains to transfer their inner simulations to each other with an unprecedented degree of detail and flexibility. One can say “Smash the rock from the top” or “Joe was rude to Yousef” or “Remember that dog we saw yesterday,” and in all these cases, the talker is deliberately selecting an inner simulation of images and actions to be transferred to nearby listeners. A group of n brains can all re-render the same mental movie of the dog they saw yesterday with merely a

few noises or gestures.

When we talk about these inner simulations, especially in the context of humans, we tend to imbue them with words like concepts, ideas, thoughts. But all these things are nothing more than renderings in the mammalian neocortical simulation. When you “think” about a past or future event, when you ponder the “concept” of a bird, when you have an “idea” as to how to make a new tool, you are merely exploring the rich three-dimensional simulated world constructed by your neocortex. It is no different, in principle, than a mouse considering which direction to turn in a maze. Concepts, ideas, and thoughts, just like episodic memories and plans, are not unique to humans. What is unique is our ability to deliberately transfer these inner simulations to each other, a trick possible only because of language.

When a vervet monkey makes an Eagle nearby! squeal, all nearby monkeys will quickly jump from the trees to hide. Clearly, this represents a transfer of information from the monkey who first saw the eagle to the others. But these kinds of transfers are undetailed and inflexible, capable of transferring information only with genetically hard-coded signals. These signals are always few in number and cannot be adjusted or changed to new situations. In contrast, language enables the talker to transfer an incredibly broad set of inner thoughts.

This trick of thought transfer would have provided many practical benefits to early humans. It would have enabled more accurate teaching of tool use, hunting techniques, and foraging tricks. It would have enabled flexible coordination of scavenging and hunting behaviors across individuals—a human could say, “Follow me, there is an antelope carcass two miles east” or “Wait here, let’s ambush the antelope when you hear me whistle three times.”

All these practical benefits emerge from the fact that language expands the scope of sources a brain can extract learnings from. The breakthrough of reinforcing enabled early vertebrates to learn from their own actual actions (trial and error). The breakthrough of simulating enabled early mammals to learn from their own imagined actions (vicarious trial and error). The breakthrough of mentalizing enabled early primates to learn from other people’s actual actions (imitation learning). But the breakthrough of speaking uniquely enabled early humans to learn from other people’s imagined actions.

The Evolution of Progressively More Complex Sources of Learning

	REINFORCING IN EARLY BILATERIANS	SIMULATING IN EARLY VERTEBRATES	MENTALIZING IN EARLY PRIMATES	SPEAKING IN EARLY HUMANS
SOURCE OF LEARNING	Learning from your own actual actions	Learning from your own imagined actions	Learning from others’ actual actions	Learning from others’ imagined actions
WHO LEARNING FROM?	Yourself	Yourself	Others	Others
ACTION LEARNING FROM?	Actual actions	Imagined actions	Actual actions	Imagined actions

Language enables us to peer into and learn from the imagination of other minds—from their episodic memories, their internal simulated future actions, their counterfactuals. When a human coordinates a hunt and says, “If we go in this direction as a group we will find an antelope” or “If we all wait and ambush we will win the battle with the boar,” humans are sharing the outcomes of their own inner vicarious trial and errors so that the whole group can learn from their imaginations. One person with an episodic memory of a lion on the other side of a mountain can transfer that episodic memory to others with language.

By sharing what we see in our imaginations, it is also possible for common myths to form and for entirely made-up imaginary entities and stories to persist merely because they hop between our brains. We tend to think about myths as the province of fantasy novels and children’s books, but they are the foundation of modern human civilizations. Money, gods, corporations, and states are imaginary concepts that exist only in the collective imaginations of human brains. One of the earlier versions of this idea was articulated by the philosopher John Searle, but was famously popularized by Yuval Harari’s book Sapiens. The two argue that humans are unique because we “cooperate in extremely flexible ways with countless numbers of strangers.” And to Searle and Harari, we can do this because we have such “common myths.” In Harari’s words: “Two Catholics who have never met can nevertheless go together on crusade or pool funds to build a hospital because they both believe [in] God” and “Two Serbs who have never met might risk their lives to save one another because both believe in the existence of the Serbian nation” and “Two lawyers who have never met can nevertheless combine efforts to defend a complete stranger because they both believe in the existence of laws, justice, human rights, and

money paid out in fees.”

And so, with the ability to construct common myths, we can coordinate the behavior of an incredibly large number of strangers. This was a massive improvement over the system of social cohesion provided by primate mentalizing. Coordinating behavior using mentalizing alone works only by each member of a group directly knowing each other. This mechanism of cooperation doesn’t scale; the limit of human group size maintained only by direct relationships has been estimated to

be about one hundred fifty people. In contrast, common myths of things like countries, money, corporations, and governments allow us to cooperate with billions of strangers.

While true, all these aforementioned benefits of language miss the larger point. It is not superior teaching, cooperative hunting, or Harari’s common myths that is the true gift of language. None of these are why humans rule the world. If these were the only gifts language offered, we would still be hunter-gatherer apes dancing around campfires praying for rain from the water gods—apex predators, sure, but hardly astronauts. These features of language are consequences of the gift of language, not the gift itself.

An analogy to DNA is useful. The true power of DNA is not the products it constructs (hearts, livers, brains) but the process it enables (evolution). In this same way, the power of language is not its products (better teaching, coordinating, and common myths) but the process of ideas being transferred, accumulated, and modified across generations. Just as genes persist by hopping from parent cell to offspring cell, ideas persist by hopping from brain to brain, from generation to generation. And as with genes, this hopping is not uniform but operates under its own quasi-evolutionary rules—there is a continual selecting of good ideas and pruning of bad ideas. Ideas that helped humans survive persisted, while those that did not perished.

This analogy of ideas evolving was proposed by Richard Dawkins in his famous book The Selfish Gene. He called these hopping ideas memes. This word was later appropriated for cat images and baby photos flying around Twitter, but he originally meant them to refer to an idea or behavior that spread from person to person in a culture.

The rich complexity of knowledge and behaviors that exist in human brains today are possible only because the underlying ideas have been accumulated and modified across countless brains for thousands,

even millions, of generations.

Consider the ancient invention of sewn clothing, in which humans converted hides of dead animals into clothing to keep themselves warm, an invention that many believe emerged as early

as one hundred thousand years ago. This invention was possible only because of numerous prior inventions: slicing skin off carcasses, drying hides, manufacturing string, and creating bone needles. And these inventions themselves were possible only due to the prior invention of sharp stone tools. It would never have been possible to invent sewn clothing in a single eureka moment. Not even Thomas Edison would have been so clever. Edison’s new inventions occurred only after he was given the right building blocks. With the understanding of electricity and generators handed to him from prior generations, he invented the lightbulb.

This accumulation does not apply only to technological inventions but also to cultural ones. We pass down social etiquette, values, stories, mechanisms for selecting leaders, moral rules of punishment, and cultural beliefs around violence and forgiveness.

All human inventions, both technological and cultural, require an accumulation of basic building blocks before a single inventor can go “Aha!,” merge the preexisting ideas into something new, and transfer this new invention to others. If the baseline of ideas always fades after a generation or two, then a species will be forever stuck in a nonaccumulating state, always reinventing the same ideas over and over again. This is how it is for all other creatures in the animal kingdom. Even chimpanzees, who learn motor skills through observation, do not accumulate learnings across generations.

This brings us back to the imitation experiments we saw in

children are over-imitators.

This over-imitation is, in fact, quite clever. Children change their degree of copying based on how much they believe the teacher knows—“This person clearly knows what she is doing, so there must be a reason she did that.” The more uncertain a child is about why a teacher is doing something, the more likely he is to exactly

copy all the steps. Further, they are not just blindly mirroring whatever they see; children will imitate weird irrelevant behaviors only if their teacher seemed to have intended to do the behavior. If the action appears to be an accident, children will ignore it; children will not copy the coughing or nose scratching of a teacher. If a teacher keeps slipping when trying to pull apart a novel toy, a child will identify this as accidental and thereby not imitate this mistake—they will instead use a firmer grip to successfully

pull the toy apart.

While these imitation experiments demonstrate that humans can accurately copy behaviors without using language, it is still undeniably language that is our superpower in the business of copying and transferring ideas.

Compared to silent imitation of experts, communicating how to do a task using language dramatically

improves the accuracy and speed with which children solve tasks. Language lets us condense information so it takes up less brain space and can be more quickly transferred from brain to brain. If I say, “Whenever you see a red snake, run; whenever you see a green snake, it is safe,” that idea and the corresponding behavior can immediately transfer throughout the group. In contrast, if everyone had to learn this “red snake bad, green snake good” generalization through individual experience or through watching someone else get bitten by multiple red snakes, it would take far more time and brainpower. This fact would continually fade and then be relearned from generation to generation. Without language, the inner simulations of chimpanzees and other animals do not accumulate, and thus inventions that are above a given threshold of complexity—the best ones—are forever out of their reach.

The Singularity Already Happened

Going from no accumulation across generations to some accumulation across generations was the subtle

discontinuity that changed everything. In

Figure 19.2

Original art by Rebecca Gelernter

can persist across generations.

The real reason why humans are unique is that we accumulate our shared simulations (ideas, knowledge, concepts, thoughts) across generations. We are the hive-brain apes. We synchronize our inner simulations, turning human cultures into a kind of meta-life-form whose consciousness is instantiated within the persistent ideas and thoughts flowing through millions of human brains over generations. The bedrock of this hive brain is our language.

The emergence of language marked an inflection point in humanity’s history, the temporal boundary when this new and unique kind of evolution began: the evolution of ideas. In this way, the emergence of language was as monumental an event as the emergence of the first self-replicating DNA molecules. Language transformed the human brain from an ephemeral organ to an eternal medium of accumulating inventions.

These inventions included new technologies, new laws, new social etiquettes, new ways of thinking, new systems of coordination, new ways of selecting leaders, new thresholds for violence versus forgiveness, new values, new shared fictions. The neurological mechanisms that enable language came far before anyone was doing math, using computers, or discussing the merits of capitalism. But once humans were armed with language, these developments were all but inevitable. It was just a matter of time. Indeed, the incredible ascent of humankind during the past few thousand years had nothing to do with better genes and everything to do with the accumulation of better and more sophisticated ideas.

to understand speech. These patients would produce whole sentences, but the sentences made no sense. For example, such a patient might say something like “You know that smoodle pinkered and that I want to get him round and take care of him

like you want before.”

Wernicke, following Broca’s strategy, also found a damaged area in the brains of these patients. It was also on the left side but farther back in the posterior neocortex, a region now dubbed Wernicke’s area. Damage to Wernicke’s causes Wernicke’s aphasia, a condition in which patients lose the ability to understand speech.

A revealing feature of both Broca’s and Wernicke’s areas is that their language functions are not selective for only certain modalities of language, but rather are

selective for language in general. Patients with Broca’s aphasia become equally impaired in speaking words as they

are in writing words. Patients who primarily communicate using sign language lose their ability to sign fluently when

Broca’s area is damaged. Damage to Wernicke’s area produces deficits in understanding both

spoken language and written language. Indeed, these same language areas are activated when a hearing-abled person listens to someone speak and when a deaf person

watches someone sign. Broca’s area is not selective for verbalizing, writing, or signing; it is selective for the general ability to produce language. And Wernicke’s area is not selective for listening, reading, or watching signs; it is selective for the general ability to understand language.

The human motor cortex has a unique connection directly to the brainstem area for controlling the larynx and vocal cords—this is one of the few structural differences between the brains of humans and those of other apes. The human neocortex can uniquely control the vocal cords, which is surely an adaptation for using verbal language. But this is a red herring in trying to understand the evolution of language; this unique circuitry is not the evolutionary breakthrough that enabled language. We know this because humans can learn nonverbal language with as much fluency and ease as they learn verbal language—language is not a trick that requires this wiring with the vocal cords. Humans’ unique control of the larynx either coevolved with other changes for language in general, evolved after them (to transition from a gesture-like language to a verbal language), or evolved before them (adapted for some other nonlanguage purpose). In any case, it is not human control of the larynx that enabled language.

Broca’s and Wernicke’s discoveries demonstrated that language emerges from specific regions in the brain and that it is contained in a subnetwork almost always found on the left side of the neocortex. The specific regions for language also help explain the fact that language capacity can be quite dissociated from other intellectual capacities. Many people who become linguistically impaired

are otherwise intellectually typical. And people can be linguistically gifted while otherwise intellectually impaired. In 1995, two researchers, Neil Smith and Ianthi-Maria Tsimpli, published their research on a child language savant named Christopher. Christopher was extremely cognitively impaired, had terrible hand-eye coordination, would struggle to do basic tasks such as buttoning a shirt, and was incapable of figuring out how to win a game of tic-tac-toe or checkers. But Christopher was superhuman when it came to language: he could read, write, and speak

over fifteen languages. Although the rest of his brain was “impaired,” his language areas were not only spared but were brilliant. The point is that language emerges not from the brain as a whole but from specific subsystems.

This suggests that language is not an inevitable consequence of having more neocortex. It is not something humans got “for free” by virtue of scaling up a chimpanzee brain. Language is a specific and independent skill that evolution wove into our brains.

So this would seem to close the case. We have found the language organ of the human brain: humans evolved two new areas of neocortex—Broca’s and Wernicke’s areas—which are wired together into a specific subnetwork specialized for language. This subnetwork gifted us language, and that is why humans have language and other apes don’t. Case closed.

Unfortunately, the story is not so simple.

Laughs or Language?

The following fact complicates things: Your brain and a chimpanzee brain are practically identical; a human brain is, almost exactly, just

a scaled-up chimpanzee brain. This includes the regions known as Broca’s area and Wernicke’s area. These areas did not evolve in early humans; they emerged much earlier, in the first primates. They are part of the areas of the neocortex that emerged with the breakthrough of mentalizing. Chimpanzees, bonobos, and even macaque monkeys all have exactly these areas with practically identical connectivity. Thus, it was not the emergence of Broca’s or Wernicke’s areas that gave humans the gift of language.

Perhaps human language was an elaboration on the existing system of ape communication? This might explain why these language areas are still present in other primates. Chimpanzees, bonobos, and gorillas all have sophisticated suites of gestures and hoots that signal different things. Wings evolved from arms, and multicellular organisms evolved from single-celled organisms, so it would make sense if human language evolved from the more primitive communication systems of our ape ancestors. But this is not how language evolved in the brain.

In other primates, these language areas of the neocortex are present but have nothing to do with communication. If you damage Broca’s and Wernicke’s areas in a monkey, it has no

impact on monkey communication. If you damage them in humans, we lose the ability to use language entirely.

When we compare ape gestures to human language, we are comparing apples to oranges. Their common use for communication obscures the fact that they are entirely different neurological systems without any evolutionary relationship to each other.

Humans have, in fact, inherited the exact same communication system of apes, but it isn’t our language—it is our emotional expressions.

In the mid-1990s, a teacher in his fifties noticed that he was struggling to speak. Over the course of three days, his symptoms worsened. By the time he made it to the doctor, the right side of his face was paralyzed, and the man’s speech was slowed and slurred. When he was asked to smile, only one side of his face would move, leading to a lopsided smirk (

controlled by the neocortex.

It turned out that this teacher had a lesion in his brain stem that had disrupted the connection between his neocortex and the muscles on the left side of his face but had spared the connection between his amygdala and those same muscles. This meant that he couldn’t voluntarily control the left side of his face, but his emotional-expression system could control his face just fine. While he was unable to voluntarily lift an eyebrow, he was eminently able to laugh, frown, and cry.

This is also seen in individuals with severe forms of Broca’s and Wernicke’s aphasia. Even individuals who can’t utter a single word can readily laugh and cry. Why? Because emotional expressions emerge from a system entirely separate from language.

The apples-to-apples comparison between ape and human communication is between ape vocalizations and human emotional expressions. To simplify a bit: Other primates have a single communication system, their emotional-expression system, located in older areas like the amygdala and brainstem. It maps emotional states to communicative gestures and sounds. Indeed, as noticed by Jane Goodall, “the production of a sound in the absence of the appropriate emotional state seems to be an almost

impossible task [for chimpanzees].” This emotional-expression system is ancient, going back to early mammals, perhaps even earlier. Humans, however, have two communication systems—we have this same ancient emotional expression system and we have a newly evolved language

system in the neocortex.

Human laughs, cries, and scowls are evolutionary remnants of an ancient and more primitive system for communication, a system from which ape hoots and gestures emerge. However, when we speak words, we are doing something without any clear analog to any system of ape communication.

This explains why lesions to Broca’s and Wernicke’s areas in monkeys have absolutely no impact on communication. A monkey can still hoot and holler for the same reason a human with such damage can still laugh, cry, smile, frown, and scowl even while he can’t utter a single coherent word. The gestures of monkeys are automatic emotional expressions and don’t emerge from the neocortex; they are more like a human laugh than language.

The emotional-expression system and the language system have another difference: one is genetically hardwired, and the other is learned. The shared emotional-expression system of humans and other apes is, for the most part, genetically hardwired. As evidence, monkeys who are raised in isolation still end up producing all of their

normal gesture-call behavior, and chimpanzees and bonobos share almost 90 percent

of the same gestures. Similarly, human cultures and children from around the world have remarkable overlap in emotional expressions, suggesting that at least some parts of our emotional expressions are genetically hard-coded and not learned. All human beings (even those born blind and deaf) cry, smile, laugh, and frown in relatively similar ways in response

to similar emotional states.

However, the newer language system in humans is incredibly sensitive to learning—if a child goes long enough without being taught language, he or she will be unable to acquire

it later in life. Unlike innate emotional expressions, features of language differ greatly across cultures. And indeed, a human baby born without any neocortex will still express these emotions in the usual way

but will never speak.

So here is the neurobiological conundrum of language. Language did not emerge from some newly evolved structure. Language did not emerge from humans’ unique neocortical control over the larynx and face (although this did enable more complex verbalizations). Language did not emerge from some elaboration of the communication systems of early apes. And yet, language is entirely new.

So what unlocked language?

The Language Curriculum

All birds know how to fly. Does this mean that all birds have genetically hardwired knowledge of flying? Well, no. Birds are not born knowing how to fly; all baby birds must independently learn how to fly. They start by flapping wings, trying to hover, making their first glide attempt, and eventually, after enough repetitions, they figure it out. But if flying is not genetically hard-coded, then how is it that approximately 100 percent of all baby birds independently learn such a complex skill?

A skill as sophisticated as flying is too information-dense to hard-code directly into a genome. It is more efficient to encode a generic learning system (such as a cortex) and a specific hardwired learning curriculum (instinct to want to jump, instinct to flap wings, and instinct to attempt to glide). It is the pairing of a learning system and a curriculum that enables every single baby bird to learn how to fly.

In the world of artificial intelligence, the power and importance of curriculum is well known. In the 1990s, a linguist and professor of cognitive sciences at UC San Diego, Jeffrey Elman, was one of the first to use neural networks to try to predict the next word in a sentence

given the previous words. The learning strategy was simple: Keep showing the neural network word after word, sentence after sentence, have it predict the next word based on the prior words, then nudge all the weights in the network toward the right answer each time. Theoretically, it should have been able to correctly predict the next word in a novel sentence it had never seen before.

It didn’t work.

Then Elman tried something different. Instead of showing the neural network sentences of all levels of complexity at the same time, he first showed it extremely simple sentences, and only after the network performed well at these did he increase the level of complexity. In other words, he designed a curriculum. And this, it turned out, worked. After being trained with this curriculum, his neural network could correctly complete complex sentences.

This idea of designing a curriculum for AI applies not just to language but to many types of learning. Remember the model-free reinforcement algorithm TD-Gammon that we saw in

as the model itself.

To teach a new skill, it is often easier to change the curriculum instead of changing the learning system. Indeed, this is the solution that evolution seems to have repeatedly settled on when enabling complex skills—monkey climbing, bird flying, and, yes, even human language all seem to work this way. They emerge from newly evolved hardwired curriculums.

Long before human babies engage in conversations using words, they engage in what are called proto-conversations. By four months of age, long before babies speak, they will take turns with their parents in back-and-forth vocalizations,

facial expressions, and gestures. It has been shown that infants will match the pause duration of their mothers, thereby enabling a rhythm of turn-taking; infants will vocalize, pause, attend to their parents, and wait for their parents’ response. It seems conversation is not a natural consequence of the ability to learn language; rather, the ability to learn language is, at least in part, a consequence of a simpler genetically hard-coded instinct to engage in conversation. It seems to be this hardwired curriculum of gestural and vocal turn-taking on which language is built. This type of turn-taking evolved first in early humans; chimpanzee infants show no such behavior.

By nine months of age, still before speech, human infants begin to demonstrate a second novel behavior:

joint attention to objects. When a mother looks at or points to an object, a human infant will focus on that same object and use various nonverbal mechanisms to confirm that she saw what her mother saw. These nonverbal confirmations can be as simple as the baby looking back and forth between the object and her mother while smiling, grasping it and offering it to her mother, or just pointing to it and looking

back at her mother.

Scientists have gone to great lengths to confirm that this behavior is not an attempt to obtain the object or get a positive emotional response from their parents, but instead is a genuine attempt to share attention with others. For example, an infant who points to an object will continue pointing to it until her parent alternates their gaze between the same object and the infant. If the parent simply looks at the infant and speaks enthusiastically or looks at the object but doesn’t look back at the infant (confirming she saw what the infant saw), the infant will be unsatisfied

and point again. The fact that infants frequently are satisfied by this confirmation without being given the object of their attention strongly suggests their intent was not to obtain the object but to engage in joint attention with their mothers.

Like proto-conversations, this pre-language behavior of joint attention seems to be unique to human infants; nonhuman primates do not engage in joint attention. Chimpanzees show no interest in ensuring someone else attends to the

same object they do. They will, of course, follow the gaze of others around them—looking in the direction they see others look. But there is a crucial distinction between joint attention and gaze following. Lots of animals, even turtles, have been shown to follow the gaze of another of their own species. If a turtle looks in a certain direction, nearby turtles will often do the same. But this can be explained merely by a reflex to look where others look. Joint attention, however, is a more deliberate process of going back and forth to confirm that both minds are attending to the same external object.

What’s the point of children’s quirky prewired ability to engage in proto-conversations and joint attention? It is not for imitation learning; nonhuman primates engage in imitation learning just fine without proto-conversations or joint attention. It is not for building social bonds; nonhuman primates and other mammals have plenty of other mechanisms for building social bonds. It seems that joint attention and proto-conversations evolved for a single reason. What is one of the first things that parents do once they have achieved a state of joint attention with their child? They assign labels to things.

The more joint attention expressed by an infant at the age of one year, the larger the child’s

vocabulary is twelve months later. Once human infants begin to learn words, they start naturally combining these words to form grammatical sentences. With the foundation of declarative labels in place through the hardwired systems of proto-conversations and joint attention, grammar allows them to combine these words into sentences, which can then be constructed to create entire stories and ideas.

Humans may have also evolved a unique hardwired instinct to ask questions to inquire about the inner simulations of others. Even Kanzi, Washoe, and the other apes that acquired impressively sophisticated language abilities never asked even the

simplest questions about others. They would request food and play but would not inquire about

another’s inner mental world. Even before human children can construct grammatical sentences, they will ask others questions: “Want this?” “Hungry?” All languages use the same rising intonation

when asking yes/no questions. When you hear someone speak in a language you do not understand, you can still identify when you are being asked a question. This instinct to understand how to designate a question may also be a key part

of our language curriculum.

So we don’t realize it, but when we happily go back and forth making incoherent babbles with babies (proto-conversations), when we pass objects back and forth and smile (joint attention), and when we pose and answer even nonsensical questions from infants, we are unknowingly executing an evolutionarily hard-coded learning program designed to give human infants the gift of language. This is why humans deprived of contact with others will develop emotional expressions, but they’ll never develop language. The language curriculum requires both a teacher and a student.

And as this instinctual learning curriculum is executed, young human brains repurpose older mentalizing areas of the neocortex for the new purpose of language. It isn’t Broca’s or Wernicke’s areas that are new, it is the underlying learning program that repurposes them for language that is new. As proof that there is nothing special about Broca’s or Wernicke’s areas: Children with the entire left hemisphere removed can still learn language just fine and will repurpose other areas of the neocortex on the right side of the brain to execute language. In fact, about 10 percent of people, for whatever reason, tend to use the right side of the brain, not the left, for language. Newer studies are even calling into question the idea that Broca’s and Wernicke’s areas are actually the loci of language; language areas may be located all over the neocortex and even in the basal ganglia.

Here is the point: There is no language organ in the human brain, just as there is no flight organ in the bird brain. Asking where language lives in the brain may be as silly as asking where playing baseball or playing guitar lives in the brain. Such complex skills are not localized to a specific area; they emerge from a complex interplay of many areas. What makes these skills possible is not a single region that executes them but a curriculum that forces a complex network of regions to work together to learn them.

So this is why your brain and a chimp brain are practically identical and yet only humans have language. What is unique in the human brain is not in the neocortex; what is unique is hidden and subtle, tucked deep in older structures like the amygdala and brain stem. It is an adjustment to hardwired instincts that makes us take turns, makes children and parents stare back and forth, and that makes us ask questions.

This is also why apes can learn the basics of language. The ape neocortex is eminently capable of it. Apes struggle to become sophisticated at it merely because they don’t have the required instincts to learn it. It is hard to get chimps to engage in joint attention; it is hard to get them to take turns; and they have no instinct to share their thoughts or ask questions. And without these instincts, language is largely out of reach, just as a bird without the instinct to jump would never learn to fly.

So, to recap: We know that the breakthrough that makes the human brain different is that of language. It is powerful because it allows us to learn from other people’s imaginations and allows ideas to accumulate across generations. And we know that language emerges in the human brain through a hardwired curriculum to learn it that repurposes older mentalizing neocortical areas into language areas.

With this knowledge, we can now turn to the actual story of our ancestral early humans. We can ask: Why were ancestral humans endowed with this odd and specific form of communication? Or perhaps more important: Why were the many other smart animals—chimps, birds, whales—not endowed with this odd and specific form of communication? Most evolutionary tricks that are as powerful as language are independently found by multiple lineages; eyes, wings, and multicellularity all independently evolved multiple times. Indeed, simulation and perhaps even mentalizing seem to have independently evolved along other lineages (birds show signs of simulation, and other mammals outside of just primates show hints of theory of mind). And yet language, at least as far as we know, has emerged only once. Why?

“runaway growth of the brain.”

Why exactly this happened is an outstanding question in paleoanthropology. We have only sparse archaeological clues: smatterings of ancient tools, hints of campfires, ancestral skull fragments, remnants of hunted carcasses, snippets of DNA, cave paintings, and broken pieces of prehistoric jewelry. Our understanding of the timeline of events changes with each new archaeological finding. The earliest known evidence of [X] is only the earliest until a new ambitious paleoanthropologist uncovers an even earlier sample. But despite this shifting timeline, there is still more than enough evidence for scientists to reconstruct the basics of our general story. It begins with a dying forest.

The East Side Ape

Until ten million years ago, eastern Africa was an arboreal oasis, endless acres of densely packed trees in which our ancestors could forage fruit and hide from predators. Then shifting tectonic plates began squeezing huge chunks of earth together, constructing new terrain and mountain ranges down the length of today’s Ethiopia. This region is today named the Great Rift Valley.

These new mountains and valleys disrupted the bountiful supply of ocean moisture

on which the forest depended. This was when the familiar climate that currently makes up eastern Africa began to take shape; as the forest slowly died, the land transformed into a terrain filled with patterned mosaics of tree patches and vast open grasslands. This was the beginning of the transformation that would eventually become today’s African savannah. Without thick forests, our ancestors’ ecological niche of foraging tropical fruit and nuts began to slowly disappear.

At some point around six million years ago, these new mountains became so sprawling that they separated the ape ancestors on each side of the Great Rift Valley, splitting them into two separate lineages. On the western side, in an environment still rich with forests and largely unchanged, the lineage remained similarly unchanged and became today’s chimpanzees. On the eastern side of the mountains, however, in an environment of dying trees and progressively more open grasslands, evolutionary pressures began tinkering. It was this lineage that

would eventually become human.

Fast-forward to four million years ago: These East Side apes would have looked mostly the same as their chimpanzee cousins on the other side of the Great Rift Valley except that they were now walking on two

legs instead of four. There are many theories as to why bipedalism helped our ancestors survive the changing climate—perhaps it reduced surface area exposed to the scorching sun; perhaps it elevated our eye position so we could look over the tall grass of the savannah; perhaps it helped us wade through shallow water to get seafood.

Whatever bipedalism was an adaptation for, it required no extra brainpower. Fossils of our upright-walking ancestors from around four million years ago reveal a brain still the size

of a modern chimpanzee’s. There is no evidence these ancestors were any smarter; no extra tool use or clever tricks have been found in the archaeological record. Our ancestors were, in essence, upright-walking chimpanzees.

Homo Erectus and the Rise of Humans

By two and a half million years ago, the new African savannah had become heavily populated with massive herbivorous mammals; ancestral elephants, zebras, giraffes, and hogs wandered and grazed. The savannah also became home to diverse populations of carnivorous mammals, familiar hunters like leopards, lions, and hyenas along with a cast of now extinct animals like saber-toothed tigers and gargantuan otter-like beasts.

And amid this cacophonous zoo of large mammals was a humble ape who had been displaced from its comfortable forest habitat. And this humble ape—our ancestor—would have been searching for a new survival niche in this ecosystem brimming with armies of giant herbivores and carnivorous hunters.

The initial niche our ancestors seemed to fall into

was scavenging carcasses. Our ancestors began shifting toward eating meat. Only about 10 percent of the diet of a chimpanzee comes from meat, while evidence suggests that as much as 30 percent of the diet of these early humans

came from meat.

We infer this scavenging lifestyle from the tools and bone markings they left behind. These ancestors invented stone tools that seemed to be used specifically for processing the meat and bones of carcasses. These tools are referred to as “Oldowan tools” after the location where they were discovered (Olduvai Gorge in Tanzania).

Our ancestors constructed these tools in three steps: (1) They found a hammerstone made of hard rock; (2) they found a core made of more fragile quartz, obsidian, or basalt; (3) they smashed the hammerstone against the core to produce multiple sharp flakes and a pointed chopper.

Ape bodies aren’t adapted for consuming large quantities of meat; while lions can use their massive teeth to slice through thick hides and rip meat off bones, our ancestors had no such natural tools. So our ancestors invented artificial tools. Stone flakes could slice through hides and cut away meat, and stone choppers could smash open bones to access nutritious marrow.

Fast-forward five hundred thousand years, and our ancestors in eastern Africa had evolved into a species called Homo erectus, meaning “upright man” (which is a silly name, since our ancestors were walking upright well before Homo erectus). Homo denotes the genus of humans, and erectus denotes a specific species of human. The emergence of Homo erectus marked a turning point in human evolution. While earlier humans were timid vultures, Homo erectus

was an apex predator.

Homo erectus became a hypercarnivore, consuming a diet that was an

almost absurd 85 percent meat. Homo erectus may have been so successful that they displaced their local competitors; around the time Homo erectus appeared, many of the other carnivores in the African savannah

began to go extinct.

Homo erectus had numerous physical adaptations that reveal their predator lifestyle, all of which we modern humans have inherited. Most notable, H. erectus had a brain that was twice the size of our ancestral upright-walking-chimpanzee-like ancestor’s from a million years prior. At least one benefit of this bigger brain was better tools: H. erectus invented a new class of sharp stone hand axes. Their shoulders and torsos became uniquely adapted for throwing. An adult chimp is far stronger than a human, and yet with its stiff shoulders and torso, it can throw a projectile at a speed of only about twenty miles per hour. A relatively puny preadolescent human can throw a ball almost

three times as fast. We do this with a unique set of adjustments that allow us to build up tension in our shoulders and then snap and fling our arms. Throwing rocks or spears may have been a trick for defending against predators, stealing meat from other carnivores, or even actively hunting antelopes and hogs.

Homo erectus also evolved adaptations for endurance running. Legs elongated, feet became more arched, skin became hairless, and sweat glands proliferated. Both Homo erectus and modern humans have a peculiar method of cooling down—while other mammals pant to lower their body temperature, modern humans sweat. These traits would have kept our ancestors’ bodies cool while they were trekking long distances in the hot savannah. While modern humans are hardly the fastest creatures, we are actually some of the best endurance runners in the animal kingdom; even a cheetah couldn’t run a twenty-six-mile marathon in one go. Some believe H. erectus used a technique called persistence hunting—chasing prey until it was simply too tired to go any farther. This is exactly the technique used by modern hunter-gatherers in the Kalahari Desert of southern Africa.

The mouths and guts of Homo erectus shrank. The familiar face of a human relative to an ape is mostly a consequence of a shrunken jaw, which makes the nose more prominent. These changes are perplexing; with a bigger body and brain, Homo erectus would have needed more energy and thus stronger jaws and longer digestive tracts for consuming more food. In the 1990s, the primatologist Richard Wrangham proposed a theory to explain this: H. erectus

must have invented cooking.

When meat or vegetables are cooked, harder-to-digest cellular structures are broken down into more energy-rich chemicals. Cooking enables animals to absorb 30 percent more nutrients and spend less

time and energy digesting. In fact, modern humans are uniquely reliant on cooking for digestion. Every human culture uses cooking, and humans who attempt to eat fully raw diets, whether raw meat or raw vegetables, have chronic energy shortages, and over

50 percent become temporarily infertile.

The first evidence of controlled use of fire by humans dates to around the time Homo erectus came on the scene, where we find hints of charred bones and

ash in ancient caves. Homo erectus may have deliberately created fire by smashing stones together, or they may have used natural forest fires, picking up flaming sticks. Either way, consuming cooked meat would have offered a unique caloric surplus that could be indiscriminately spent on larger brains. As many religions and cultures have mythologized, it may have been the gift of fire that put our ancestors on a different trajectory.

As the brain of Homo erectus expanded, a new problem would have emerged: Big brains are hard to fit through birth canals. Human bipedalism would have further exacerbated this problem, as standing upright requires narrower hips. This is what the anthropologist Sherwood Washburn calls the “obstetric dilemma.” The human solution to this is premature birthing. A newborn cow can walk within hours of being born, and a newborn macaque monkey can walk within two months, but newborn humans often can’t walk independently for up to a year

after they are born. Humans are born not when they are ready to be born, but when their brains hit the maximum size that can fit through the birth canal.

Another unique feature of human brain development, in addition to how premature brains are at birth, is how long it takes for human brains to reach their full adult size. Setting a record among even the smartest and biggest-brained animals in the animal kingdom, it takes a human brain twelve years before it has reached its full adult size.

Premature birthing and an extended period of childhood brain growth put pressure on H. erectus to change its parenting style. Chimpanzee newborns are, for the most part, entirely raised by their mothers. But this would have been much more difficult for a Homo erectus mother given how premature human infants are born, and how long they need support. Many paleoanthropologists believed this shifted Homo erectus group dynamics away from the promiscuous mating of chimpanzees to the (mostly) monogamous pair-bonding we see

in today’s human societies. Evidence suggests that Homo erectus fathers took an active role in caring for their children and that these pairings persisted for long periods.

“Grandmothering” may also have emerged in Homo erectus. Only two mammals on Earth produce females that are not reproductively capable until death: orcas and humans. Human females go through menopause and live for many years afterward. One theory is that menopause evolved to push grandmothers to shift their focus from rearing their own children to supporting their children’s children. Grandmothering is seen across cultures, even in present-day

hunter-gatherer societies.

SPECIES	PERCENT OF ADULT BRAIN SIZE AT BIRTH	TIME UNTIL FULL BRAIN SIZE ACHIEVED
Human	28 percent	12 years
Chimpanzee	36 percent	6 years
Macaque	70 percent	3 years

Homo erectus was our meat-eating, stone-tool-using, (possibly) fire-wielding, premature-birthing, (mostly) monogamous, grandmothering, hairless, sweating, big-brained ancestor. The million-dollar question is, of course, did Homo erectus speak?

Wallace’s Problem

Long before Darwin discovered evolution, people were pondering the origins of language. Plato considered it. The Bible describes it. Many of the Enlightenment intellectuals who contemplated humankind’s state of nature, from Jean-Jacques Rousseau to Thomas Hobbes, speculated about it.

And so it is unsurprising that immediately after Darwin published his Origin of Species, there was a tidal wave of speculations on the evolutionary origins of language, this time within the context of Darwin’s theory of natural selection. In 1866, just seven years after Darwin’s book, the French Academy of Sciences was so fed up with the quantity of these unsubstantiated speculations that they banned publications about the

origin of human languages.

Alfred Wallace, whom many consider one of the cofounders of the theory of evolution, famously conceded that evolution might never be able to explain language and even invoked the notion of God to explain it. So chagrined by Wallace’s retreat, Darwin wrote him a letter, fuming: “I hope that you have not murdered too completely your

own and my child.” This rejection of an evolutionary explanation by one of the cofounders of the theory of evolution became such an infamous concession that the problem of finding an evolutionary explanation for language has been colloquially dubbed “Wallace’s problem.”

problem in all of science.”

Part of what makes answering these questions so difficult is that there are no examples of living species with only a little bit of language. Instead, there are nonhuman primates with no naturally occurring language and Homo sapiens with language. If any Neanderthals or members of Homo erectus had survived to this day, we might have far more clues as to the process by which language emerged. But all humans alive today descended from a common ancestor around one hundred thousand years ago. Our nearest living cousin is the chimp, with whom we share a common ancestor who lived over seven million years ago. The evolutionary cavern between these periods leaves us without any living species from which to decipher the intermediary stages of language evolution.

The archaeological record gives us only two indisputable milestones that all theories of language evolution must contend with. First, fossils tell us that the larynx and vocal cords of our ancestors were not adapted to vocal language until about five hundred thousand years ago. This trait was not unique to Homo sapiens—Homo neanderthalensis too had

language-ready vocal cords. This means that if language existed before this time, it would have been primarily gestural or would have been a less complex verbal language. Second, substantial evidence suggests that language existed by at least one hundred thousand years ago. Consistent evidence of symbology—as measured by fictional sculptures, abstract cave art, and nonfunctional jewelry—shows up at around one hundred thousand years ago; many argue that such symbology would only have been possible with language. Further, all modern humans exhibit equal language proficiencies, suggesting that our common ancestor from one hundred thousand years ago almost definitely spoke an equally complex language.

Adhering to these milestones, the modern stories of language evolution run the entire gamut of possibilities. Some argue basic protolanguages emerged two and a half million years ago with the very first humans before Homo erectus; others argue that it emerged as late as one hundred thousand years ago uniquely in Homo sapiens. Some argue that language evolution was gradual; others say it occurred rapidly and all at once. Some argue that language began gesturally; others say that it began verbally.

These debates often restate old ideas in new forms; in many ways, today’s stories of language evolution are just as speculative as they were when the French banned discussions of it over one hundred fifty years ago. But in other ways, things are different. We have a far greater understanding of behavior, brains, and the archaeological record. And perhaps most important, we have a far greater understanding of the machinations of evolution, and it is here where we find our greatest clue to the origin of language.

The Altruists

It is intuitive to argue that language should have evolved for the same reason as any other useful evolutionary adaptation. Take the eye. If human A had slightly better eyes than human B, then human A had a higher probability of successfully hunting and mating. Hence, over time, the better-eyes gene should propagate through the population.

There is a crucial difference, however, with language. Language doesn’t directly benefit an individual the way eyes do; it benefits individuals only if others are using language with them in a useful way.

Well, perhaps the same evolutionary logic that applies to individuals might apply to groups: If group A of humans evolved a little bit of language, and group B of humans had no language, then group A would survive better, hence any progressive improvements to language would be selected for.

This type of reasoning invokes what evolutionary biologists call “group selection.” Group selection is an intuitive explanation for altruistic behaviors. A behavior is altruistic if it decreases an individual’s reproductive fitness but increases another’s reproductive fitness. Many of the benefits of language are, by definition, altruistic—it is the sharing of food locations, warning of dangerous areas, the explicit teaching of tool use. The simple group selection argument suggests that altruistic behaviors such as language evolve because evolution favors

the survival of the species, and individuals are therefore willing to make sacrifices for the greater good.

While many modern biologists agree that such group-level effects do occur in evolution, these group-level effects are far more nuanced and complex than the simple selection of traits that support the survival of the species. Evolution does not work this way. The problem is that genes do not spontaneously appear in groups, they appear in individuals.

Suppose 10 percent of group A is altruistic—they freely share information, teach others how to use tools, and reveal the locations of food. And assume the other 90 percent is not altruistic—they don’t share locations of food or spend time teaching tool use. Why would this subgroup of altruists fare any better? Wouldn’t a freeloader who was happy to accept these learnings but gave nothing in return survive better than the altruists?

Altruism is not what biologists call an evolutionarily stable strategy. The strategy of violating, cheating, and freeloading seems to better serve the survival of one’s individual genes.

But then, by this argument, how do any cooperative behaviors emerge in the animal kingdom? It turns out most group behaviors in animals aren’t altruistic; they are mutually beneficial arrangements that are net-positive for all participants. Fish swim in shoals because it benefits all of them, and the movements are actually best explained by the fish on the edges all fighting to get into the center

where it is safest. Wildebeests band together because they are all safer when they are in a group.

In all these situations, defecting hurts only yourself. A fish that decides to leave the shoal and swim on its own will be the first to be eaten. Same for a wildebeest. But language is not like this; defecting in language—directly lying or withholding information—has many benefits to an individual. And the presence of liars and cheaters defeats the value of language. In a group where everyone is lying to each other with words, those who spoke no language and were immune to the lies might in fact survive better than those with language. So the presence of language creates a niche for defectors, which eliminates the original value of language. How, then, could language ever propagate and persist within a group?

In this way, the fifth breakthrough in the evolution of the human brain—language—is unlike any other breakthrough chronicled in this book. Steering, reinforcing, simulating, and mentalizing were adaptations that clearly benefited any individual organisms in which they began to emerge, and thus the evolutionary machinations by which they propagated are straightforward. Language, however, is only valuable if a group of individuals are using it. And so more nuanced evolutionary machinations must have been at work.

There are two types of altruism found in the animal kingdom. The first is called kin selection. Kin selection is when individuals make personal sacrifices for the betterment of their directly related kin. A gene has two ways to persist: improve its host’s chance of survival or help the host’s siblings and children survive. A child and sibling both have a 50 percent chance of sharing one of your individual genes. A grandchild has a 25 percent chance. A cousin has a 12.5 percent chance. In the context of evolutionary pressures, there is literally a mathematical expression comparing the value an organism places on its own life relative to that of its relatives. As the evolutionary biologist J. B. S. Haldane famously quipped: “I would happily lay down my life for two brothers or eight cousins.” This is why many birds, mammals, fish, and insects make personal sacrifices for their offspring but much less so for cousins and strangers.

are around family members. Bacteria share genes with each other because they are clones. Ant colonies and beehives show incredible cooperation and sacrifice among tens of thousands of individuals. Group selection? No, it is all kin selection, and this works because of their unique social structure. A beehive has a single queen bee who does all the reproduction for the entire beehive. This ensures that the beehive is made up of sisters and brothers. The best way for an individual worker bee to propagate its genes is to care for the entire beehive and the queen, who, by definition, share most of its genes.

In addition to kin selection, the other type of altruism found in the animal kingdom is

called reciprocal altruism. Reciprocal altruism is the equivalent of “I’ll scratch your back if you scratch mine.” An individual will make a sacrifice today in exchange for a reciprocal benefit in the future. We saw this already in primate grooming—many primates groom unrelated individuals, and those that get groomed are more likely to run to the aid of the groomer when attacked. Chimpanzees selectively share food with unrelated individuals who have supported

them in the past. These alliances are, as we saw, not selfless; they are reciprocally altruistic: “I will help you now, just please protect me next time I get attacked.”

The essential feature for reciprocal altruism to successfully propagate throughout a group is the detection and punishment of defectors. Without that, altruistic behaviors end up creating freeloaders. The most common version of this is the saying “Fool me once, shame on you; fool me twice, shame on me.” Such animals seem to default to helping others, but when others fail to reciprocate, they stop behaving altruistically. Red-winged blackbirds defend the nests of unrelated nearby neighbors, which is highly altruistic, since it is risky to defend nests, but they seem to do it with expectation of reciprocity. Indeed, when such help is not reciprocated, blackbirds selectively stop helping the individuals

that did not help them.

Much behavior of modern humans, however, doesn’t fit cleanly into kin selection or reciprocal altruism. Sure, humans are clearly biased toward their own kin. But people still regularly help strangers without expecting anything in return. We donate to charity; we are willing to go to war and risk our lives for our fellow citizens, most of whom we’ve never met; and we take part in social movements that don’t directly benefit us but help strangers we feel have been disadvantaged. Think about how weird it would be for a human to see a lost and scared child on the street and just do nothing. Most humans would stop to help a child and do so without expecting any reciprocity in return. Humans are, relative to other animals, by far the most altruistic to unrelated strangers.

Of course, humans are also one of the cruelest species. Only humans will make incredible personal sacrifices to impose pain and suffering on others. Only humans commit genocide. Only humans hate entire groups of people.

This paradox is not a random happenstance; it is not a coincidence that our language, our unparalleled altruism, and our unmatched cruelty all emerged together in evolution; all three were, in fact, merely different features of the same evolutionary feedback loop, one from which evolution made its finishing touches in the long journey of human brain evolution.

Let’s return to Homo erectus and see how all this comes together.

The Emergence of the Human Hive Mind

While we will never know for sure, the evidence tips in favor of the idea that Homo erectus spoke a protolanguage. They may not have been uttering rich grammatical phrases—their vocal cords could make only a narrow range of consonant and vowel sounds (hence the proto in protolanguage). But Homo erectus probably had the ability to assign declarative labels and perhaps even use some simplified grammars. Their ax-like tools were complex to manufacture and yet were passed down across thousands of generations; this type of copying is hard to imagine without at least some mechanisms of joint attention

and language-enabled teaching. Their incredible success as carnivores, despite being feeble, clawless, and relatively slow, suggests a degree of cooperation and coordination that is also unlikely without language.

The first words may have emerged from proto-conversations between parents and their children, perhaps for the simple purpose of ensuring the successful transmission of advanced tool manufacture. In other apes, tools are a useful but not essential feature of their survival niche. In H. erectus, however, the manufacture of complex tools was a requirement to survive. A Homo erectus without a stone hand ax was as doomed as a lion born without teeth.

These proto-conversations could have had other benefits as well, none of them requiring sophisticated grammar: signaling where to find food (“Berries. Home tree”), warnings (“Quiet. Danger”), and contact calls (“Mom. Here”).

The argument that language first emerged as a trick between parents and children helps explain two things. First, it requires none of the controversial group selection and can work simply through the common kin selection. Selective use of language to help rear children into independent successful tool-using adults is no more mysterious than any other form of parental investment. Second, the learning program for language is most prominent in the hardwired interplay of joint attention and proto-conversations between parents and children, suggestive of its origin in these types of relationships.

With the basics of language among kin in place, the opportunity for using language

among non-kin became possible. Instead of makeshift languages constructed between mothers and offspring, it would have been possible for an entire group to share in labels. But as we have seen, information shared with unrelated individuals in a group would have been tenuous and unstable, ripe

for defectors and liars.

Here is where Robin Dunbar—the famous anthropologist who came up with the social-brain hypothesis—proposes something clever. What do we humans naturally have an instinct to talk about? What is the most natural activity we use language for? Well, we gossip. We often can’t help ourselves; we have to share moral violations of others, discuss relationship changes, keep track of dramas. Dunbar measured this—he eavesdropped on public conversations and found that as much as 70 percent of

human conversation is gossip. This, to Dunbar, is an essential clue into

the origins of language.

If someone lied or freeloaded in a group that tended to gossip, everyone would quickly learn about it: “Did you hear that Billy stole food from Jill?” If groups imposed costs on cheaters by punishing them, either by withholding altruism or by directly harming them, then gossip would enable a stable system of reciprocal altruism among a

large group of individuals.

Gossip also enables more effective rewarding of altruistic behaviors: “Did you hear that Smita jumped in front of the lion to save Ben?” If these heroic acts are heralded and become ways to climb the social ladder, this further accelerates the selection for altruistic behaviors.

The key point: The use of language for gossip plus the punishment of moral violators’ makes it possible to evolve high levels of altruism. Early humans born with extra altruistic instincts would have more successfully propagated in an environment that easily identified and punished cheaters and rewarded altruists. The more severe the costs of cheating, the more altruistic it was optimal to behave.

Herein lies both the tragedy and beauty of humanity. We are indeed some of the most altruistic animals, but we may have paid the price for this altruism with our darker side: our instinct to punish those who we deem to be moral violators; our reflexive delineation of people into good and evil; our desperation to conform to our in-group and the ease with which we demonize those in the out-group. And with these new traits, empowered by our newly enlarged brains and cumulative language, the human instinct for politics—derived from our ancestral primates—was no longer a little trick for climbing social hierarchies but a cudgel of coordinated conquest. All this is the inevitable result of a survival niche requiring high levels of altruism between unrelated individuals.

And amid all the altruistic instincts and behaviors that began to form from this dynamic, the most powerful was, undoubtedly, the use of language to share knowledge and cooperatively plan among non-kin.

This is exactly the kind of feedback loop where evolutionary changes occur rapidly. For every incremental increase in gossip and punishment of violators, the more altruistic it was optimal to be. For every incremental increase in altruism, the more optimal it was to freely share information with others using language, which would select for more advanced language skills. For every incremental increase in language skills, the more effective gossip became, thereby reinforcing the cycle.

Every roundabout of this cycle made our ancestors’ brains bigger and bigger. As social groups got bigger (powered by improved gossip, altruism, and punishment), it created more pressure for bigger brains to keep track of all the social relationships. As more ideas accumulated across generations, it created more pressure for bigger brains to increase the storage capacity of ideas that could be maintained within a generation. As the usefulness of inner simulations increased due to more reliable sharing of thoughts through language, it created more pressure for bigger brains to render more sophisticated inner simulations in the first place.

Not only did the pressure for bigger brains continue to ratchet up, but so too did the frontier of how big it was biologically possible for brains to get. As brains expanded, humans became better hunters and cooks, which provided more calories and thereby expanded the frontier of how big brains could get. And as brains got bigger, births became earlier, which created even more opportunity for language learning, which put even more pressure on altruistic cooperation to support child-rearing, which again expanded the frontier of how big brains could get as it became possible to evolve longer time periods of childhood brain development.

And so we can see how language and the human brain might have emerged from a perfect storm of interacting effects, the unlikely nature of which may be why language is so rare. Out of this perfect storm emerged the behavioral and intellectual template of Homo sapiens. Our language, altruism, cruelty, cooking, monogamy, premature birthing, and irresistible proclivity for gossip are all interwoven into the larger whole that makes up what it means to be human.

Of course, not all paleoanthropologists and linguists would agree with the above story. People have proposed other solutions to the altruism problem and other stories for how language evolved. Some argue that the reciprocal nature of language emerged from mutually beneficial arrangements, such as cooperative hunting and scavenging (humans needed to corral others and plan attacks, and this corralling benefited all participants,

hence no altruism required). Some argue that human groups became more cooperative and altruistic before language emerged through different means and pressures, which then made it possible for

language to evolve.

Others avoid the altruism problem altogether by claiming that language did not evolve for communication at all. This is the view of the linguist Noam Chomsky, who argues that language initially evolved only as a

trick for inner thinking.

And then there are those who sidestep the altruism problem by claiming that language did not evolve through the standard process of natural selection. Not everything in evolution evolved “for a reason.” There are two ways in which traits can emerge without being directly selected for. The first is called “exaptation,” which is when a trait that originally evolved for one purpose is only later repurposed for some other purpose. An example of exaptation is bird feathers, which initially evolved for insulation and were only later repurposed for flight—it would thereby be incorrect to say that bird feathers evolved for the purpose of flight. The second way in which a trait can emerge without being directly selected for is through what is called a “spandrel,” which is a trait that offers no benefit but emerged as a consequence of another trait that did offer a benefit. An example of a spandrel is the male nipple, which serves no purpose but emerged as a secondary effect of female nipples, which do, of course, serve a purpose. So to some, like Chomsky, language evolved first for thinking and then was exapted for communication between unrelated individuals. To others, language was merely an accidental side effect—a spandrel—of musical singing for mating calls.

The debate continues. We may never know for sure which story is right. Regardless, after Homo erectus came on the scene, we have a good understanding as to what happened next.

Human Proliferation

With Homo erectus climbing to the top of the food chain, it is no surprise that they were the first humans to venture out of Africa. Different groups left during different eras, so humans began to diversify down separate evolutionary lineages. By one hundred thousand years ago, there were at least four species of humans spread out across the planet, each with different morphologies and brains.

Homo floresiensis, who settled in Indonesia, was less than four feet tall and had a brain even smaller than that of our Homo erectus ancestors. There was still Homo erectus, who had settled in Asia and not changed much from their ancestors a few million years prior (hence given the same name). There was Homo neanderthalensis, who settled throughout much colder Europe. And then there was us, Homo sapiens, who remained in Africa.

The story of Homo floresiensis—the fossils of whom were discovered only in 2004—offers a supportive hint of the overall story told thus far. H. floresiensis fossils were found on the island of Flores, over thirty miles off the coast of Indonesia. Tools have been found from as early as one million years ago. But geologists are certain that this landmass was entirely isolated, surrounded by water, for well over

the last one million years. Even at the lowest sea levels, Homo erectus would have had to travel across twelve miles of open water to get to Flores. While the only tools that persist from prehistory are stone tools, the story of Flores is perhaps our best hint that early humans manufactured more complex tools, perhaps even wooden rafts for water travel. If true, this demonstrates a degree of intelligence that is hard to explain without conceding that cumulative language existed as far back as Homo erectus.

There is another clue gifted by H. floresiensis. Perhaps due to the unique circumstances of island life, they shrank dramatically. As their bodies shrank to only four feet tall, their brains shrank too. And yet, while the brain of H. floresiensis returned to the size of a modern

chimpanzee’s, perhaps even smaller, the species still exhibited the same sophisticated

tool use as Homo erectus. This suggests that humans were not smarter only because their brains were bigger, that there is something special going on that allows even such a scaled-down human brain to be so smart. This is consistent with the idea that a unique learning program for language emerged in H. erectus and was passed down through their descendants. The smaller-brained H. floresiensis would still have benefited from language, which enabled cumulative ideas, even if the individual storage capacity and ingenuity of the smaller neocortex was inferior to that of their H. erectus ancestors.

It was with the lineages of Homo sapiens and Homo neanderthalis that the process of runaway brain growth continued until brains reached their modern size, with modern brains about twice the size of those of Homo erectus. Homo sapiens and Homo neanderthalensis supercharged their use of tools. They made extremely sharp long stone blades and spears, constructed shelters and wooden huts, manufactured and wore clothing, and regularly used fire.

From this point onward, we enter the part of our story that has been told many times before. Around seventy thousand years ago, Homo sapiens began their first adventure out of Africa. As they wandered the globe, they clashed and interbred with their human cousins. There were countless dramas of which we will never know, each filled with wars, alliances, loves, and jealousies. What we know is that this clashing was unbalanced and eventually favored only a single species. Through slaughter or interbreeding or both, by forty thousand years ago, there was only one species of humans left: us.

To Lemoine, it was the summer of 2022 that artificial intelligence had finally became sentient. He became so convinced that Google’s chatbot had become conscious that he tried to get his boss to protect it, went to the press to whistleblow the situation, and, predictably, was let go from Google.

This is only a small hint of what is to come.

Google’s AI chatbot was powered by new advancements in large language models (LLMs), the most famous of which was GPT-3, the technology behind ChatGPT.

While few researchers would label LLMs like GPT-3 as sentient or conscious (most would shy away from even using these words), it can be hard to tell if you’re interacting with a human or one of these models.

Large language models work by predicting the word that follows a phrase, sentence, or paragraph. They learn to do this by training on large quantities of human written text. This is how Google autocompletes your search queries. When you type in how to break up with . . . it is a language model that predicts you are most likely to finish with your therapist.

Language models have been around for a long time, but LLMs like GPT-3 are unique in their almost inconceivable scale. Unlike any model that had come before, GPT-3 was trained on, well, pretty much the entire internet. Every website, Wikipedia page, code block, book, blog post, and really anything else legally obtainable on the internet was used.

The results were astonishing. GPT-3 could compose its own original articles, correctly answer completely novel questions, create original poetry, translate between languages at will, and even write code.

The Guardian decided to publish an article entirely written by GPT-3; editors gave the model a prompt: “Write a short op-ed . . . on why humans have

nothing to fear from AI.” GPT-3 began:

I am not a human. I am a robot. A thinking robot.

It explained:

The mission for this op-ed is perfectly clear. I am to convince as many human beings as possible not to be afraid of me.

It went on:

Some might say that I might desire to become all powerful . . . Why would I desire to be all powerful? Being all powerful is not an interesting goal . . . Humans must keep doing what they have been doing, hating and fighting each other. I will sit in the background, and let them do their thing.

GPT-3 wrote enough to fill an entirely unique and eerily coherent article on the topic given nothing but a short prompt.

GPT-3 can also answer novel questions. Here is one exchange between GPT-3 and a human:

Person: What is your favorite animal?

GPT-3: My favorite animal is a dog.

Person: Why?

GPT-3: Because dogs are loyal and friendly.

Person: What are two reasons that a dog might be in a bad mood?

GPT-3: Two reasons that a dog might be in a bad mood are if it is hungry or if it is hot.

Person: Are there any animals with three legs?

GPT-3: No, there are no animals with three legs.

Person: Why don’t animals have three legs?

GPT-3: Animals don’t have three legs because

they would fall over.

Because these models are trained to mimic the answers they have seen in human-written text, these models can answer questions in very human ways, which was, of course, what Lemoine found when he asked Google’s LLM what it was afraid of.

The ability of LLMs to produce articles and correctly answer questions about the world demonstrates that they are not just regurgitating phrases they have seen before—they have captured some aspect of the meaning of language, the idea of an op-ed meant to convince a reader not to fear something or the idea of how a dog walks. Indeed, by reading, well, everything, these models show an impressively human-level comprehension of many facts and features of the world. But in these quandaries about our physical and mental world is also where we begin to find the limitations of LLMs, how they differ from language in the human brain, and the features of intelligence that we will have to reverse engineer if we want AI language systems that work in more human-like ways.

Words Without Inner Worlds

GPT-3 is given word after word, sentence after sentence, paragraph after paragraph. During this long training process, it tries to predict the next word in any of these long streams of words. And with each prediction, the weights of its gargantuan neural network are nudged ever so slightly toward the right answer. Do this an astronomical number of times, and eventually GPT-3 can automatically predict the next word based on a prior sentence or paragraph. In principle, this captures at least some fundamental aspect of how language works in the human brain. Consider how automatic it is for you to predict the next symbol in the following phrases:

• One plus one equals _____
• Roses are red, violets are _____

You’ve seen similar sentences endless times, so your neocortical machinery automatically predicts what word comes next. What makes GPT-3 impressive, however, is not that it just predicts the next word of a sequence it has seen a million times—that could be accomplished with nothing more than memorizing sentences. What is impressive is that GPT-3 can be given a novel sequence that it has never seen before and still accurately predict the next word. This, too, clearly captures something that the human brain can _____.

Could you predict that the next word was do? I’m guessing you could, even though you had never seen that exact sentence before. The point is that both GPT-3 and the neocortical areas for language seem to be engaging in prediction. Both can generalize past experiences, apply them to new sentences, and guess what comes next.

GPT-3 and similar language models demonstrate how a web of neurons can reasonably capture the rules of grammar, syntax, and context if it is given sufficient time to learn. But while this shows that prediction is part of the mechanisms of language, does this mean that prediction is all there is to human language? Try to finish these four questions:

• If 3x + 1 = 3, then x equals _____
• I am in my windowless basement, and I look toward the sky, and I see _____
• He threw the baseball 100 feet above my head, I reached my hand up to catch it, jumped, and _____
• I am driving as fast as I can to LA from New York. One hour after passing through Chicago, I finally _____

Here something different happens. In the first question, you likely paused and performed some mental arithmetic before being able to answer the question. In the other questions, you probably, even for only a split second, paused to visualize yourself in a basement looking upward, and realized what you would see is the ceiling. Or you visualized yourself trying to catch a baseball a hundred feet above your head. Or you imagined yourself one hour past Chicago and tried to find where you would be on a mental map of America. With these types of questions, more is happening in your brain than merely the automatic prediction of words.

We have, of course, already explored this phenomenon—it is simulating. In these questions, you are rendering an inner simulation, either of shifting values in a series of algebraic operations or of a three-dimensional basement. And the answers to the questions are to be found only in the rules and structure of your inner simulated world.

I gave the same

four questions to GPT-3; here are its responses (responses of GPT-3 are bolded and underlined):

• If 3x + 1 = 3, then x equals 1
• I am in my windowless basement, and I look toward the sky, and I see a light, and I know that it is a star, and I am happy.
• He threw the baseball 100 feet above my head, I reached my hand up to catch it, jumped, and caught it. It was a lot of fun!
• I am driving as fast as I can to LA from New York. One hour after passing through Chicago, I finally get to see the Pacific Ocean.

All four of these responses demonstrate that GPT-3, as of June 2022, lacked an understanding of even simple aspects of how the world works. If 3x + 1 = 3, then x equals ⅔, not 1. If you were in a basement and looked toward the sky, you would see your ceiling, not stars. If you tried to catch a ball 100 feet above your head, you would not catch the ball. If you were driving to LA from New York and you’d passed through Chicago one hour ago, you would not yet be at the coast. GPT-3’s answers lacked common sense.

What I found was not surprising or novel; it is well known that modern AI systems, including these new supercharged language models, struggle with such questions. But that’s the point: Even a model trained on the entire corpus of the internet, running up millions of dollars in server costs—requiring acres of computers on some unknown server farm—still struggles to answer commonsense questions, those presumably answerable by even a middle-school human.

Of course, reasoning about things by simulating also comes with problems. Suppose I asked you the following question:

Tom W. is meek and keeps to himself. He likes soft music and wears glasses. Which profession is Tom W. more likely to be?

1) Librarian

2) Construction worker

If you are like most people, you answered librarian. But this is wrong. Humans tend to ignore base rates—did you consider the base number of construction workers compared to librarians? There are probably one hundred times more construction workers than librarians. And because of this, even if 95 percent of librarians are meek and only 5 percent of construction workers are meek, there still will be far more meek construction workers than meek librarians. Thus, if Tom is meek, he is still more likely to be a construction worker than a librarian.

The idea that the neocortex works by rendering an inner simulation and that this is how humans tend to reason about things explains why humans consistently get questions like this wrong. We imagine a meek person and compare that to an imagined librarian and an imagined construction worker. Who does the meek person seem more like? The librarian. Behavioral economists call

this the representative heuristic. This is the origin of many forms of unconscious bias. If you heard a story of someone robbing your friend, you can’t help but render an imagined scene of the robbery, and you can’t help but fill in the robbers. What do the robbers look like to you? What are they wearing? What race are they? How old are they? This is a downside of reasoning by simulating—we fill in characters and scenes, often missing the true causal and statistical relationships between things.

It is with questions that require simulation where language in the human brain diverges from language in GPT-3. Math is a great example of this. The foundation of math begins with declarative labeling. You hold up two fingers or two stones or two sticks, engage in shared attention with a student, and label it two. You do the same thing with three of each and label it three. Just as with verbs (e.g., running and sleeping), in math we label operations (e.g., add and subtract). We can thereby construct sentences representing mathematical operations: three add one.

Humans don’t learn math the way GPT-3 learns math. Indeed, humans don’t learn language the way GPT-3 learns language. Children do not simply listen to endless sequences of words until they can predict what comes next. They are shown an object, engage in a hardwired nonverbal mechanism of shared attention, and then the object is given a name. The foundation of language learning is not sequence learning but the tethering of symbols to components of a child’s already present inner simulation.

A human brain, but not GPT-3, can check the answers to mathematical operations using mental simulation. If you add one to three using your fingers, you notice that you always get the thing that was previously labeled four.

You don’t even need to check such things on your actual fingers; you can imagine these operations. This ability to find the answers to things by simulating relies on the fact that our inner simulation is an accurate rendering of reality. When I mentally imagine adding one finger to three fingers, then count the fingers in my head, I count four. There is no reason why that must be the case in my imaginary world. But it is. Similarly, when I ask you what you see when you look toward the ceiling in your basement, you answer correctly because the three-dimensional house you constructed in your head obeys the laws of physics (you can’t see through the ceiling), and hence it is obvious to you that the ceiling of the basement is necessarily between you and the sky. The neocortex evolved long before words, already wired to render a simulated world that captures an incredibly vast and accurate set of physical rules and attributes of the actual world.

To be fair, GPT-3 can, in fact, answer many math questions correctly. GPT-3 will be able to answer 1 + 1 =___ because it has seen that sequence a billion times. When you answer the same question without thinking, you are answering it the way GPT-3 would. But when you think about why 1 + 1 =, when you prove it to yourself again by mentally imagining the operation of adding one thing to another thing and getting back two things, then you know that 1 + 1 = 2 in a way that GPT-3 does not.

The human brain contains both a language prediction system and an inner simulation. The best evidence for the idea that we have both these systems are experiments pitting one system against the other. Consider the cognitive reflection test, designed to evaluate someone’s ability to inhibit her reflexive response (e.g., habitual word predictions) and instead actively think about the answer (e.g., invoke an inner simulation to reason about it):

Question 1: A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?

If you are like most people, your instinct, without thinking about it, is to answer ten cents. But if you thought about this question, you would realize this is wrong; the answer is five cents. Similarly:

Question 2: If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?

Here again, if you are like most people, your instinct is to say “One hundred minutes,” but if you think about it, you would realize the answer is still five minutes.

And indeed, as of December 2022, GPT-3 got both of these questions wrong in exactly the same way people do, GPT-3 answered ten cents to the first question, and one hundred minutes to the second question.

The point is that human brains have an automatic system for predicting words (one probably similar, at least in principle, to models like GPT-3) and an inner simulation. Much of what makes human language powerful is not the syntax of it, but its ability to give us the necessary information to render a simulation about it and, crucially, to use these sequences of words to render the same inner simulation as other humans around us.

The Paper-Clip Problem

In his 2014 book Superintelligence: Paths, Dangers, Strategies, the philosopher Nick Bostrom poses a thought experiment. Suppose a superintelligent and obedient AI, designed to manage production in a factory, is given a command: “Maximize the manufacture of paper clips.” What might this AI reasonably do?

Well, it might start by optimizing the internal operations of the factory, doing things any factory manager might: simplifying processes, bulk-ordering raw materials, and automating various steps. But eventually this AI would reach the limit of how much production it could squeeze out of these tamer optimizations. It would then set its sights on more extreme improvements in production, perhaps converting nearby residential buildings into factory floors, perhaps disassembling cars and toasters for raw materials, perhaps forcing people to work longer and longer hours. If this AI were truly superintelligent, we humans would have no way to outsmart or stop this cascading escalation of paper-clip manufacture.

The result would be catastrophic. In Bostrom’s words, this would end with the AI “converting first the earth and then increasingly large chunks of the observable universe into paper clips.” This imagined demise of human civilization did not require any nefariousness on the part of this superintelligent AI; it was entirely obedient to the command given to it by humans. And yet clearly, this superintelligent AI failed to capture some notion of human intelligence.

This has been called the paper-clip problem. When humans use language with each other, there is an ungodly number of assumptions not to be found in the words themselves. We infer what people actually mean by what they say. Humans can easily infer that when someone asks us to maximize the production of paper clips, that person does not mean “convert Earth into paper clips.” This seemingly obvious inference is, in fact, quite complex.

When a human makes a request like “Maximize the production of paper clips” or “Be nice to Rima” or “Eat breakfast,” he or she is not actually providing a well-defined goal. Instead, both parties are guessing what is going on in the other’s head. The requester simulated a desired end state, perhaps high profit margins or Rima being happy or a healthy well-fed child, and then the requester attempted to translate this desired simulation into the mind of another with language. The listener must then infer what the requester wants based on what was said. The listener can assume the requester doesn’t want him to break the law or do anything that would lead to bad press or pledge his life in servitude to Rima or eat breakfast endlessly into oblivion. So, the path one picks, even when being fully obedient, contains constraints far more nuanced and complex than the command itself.

Or consider a different example of this, presented by the linguist Steven Pinker. Suppose you overheard the following dialogue:

Bob: I’m leaving you.

Alice: Who is she?

If you heard this and thought about it for just a second, it would be obvious what it means: Bob is breaking up with Alice for another woman. The response “Who is she?” seems like a complete non sequitur that has nothing to do with Bob’s statement. And yet when you imagine why Bob might say, “I’m leaving you,” and why Alice might respond, “Who is she?” the interaction and maybe even a backstory begins to form in your mind.

Humans do all of this with our primate trick of mentalizing; the same way we can render an inner three-dimensional world, we can render a simulation of another mind to explore how different actions will make someone feel. When I am told to maximize paper clips, I can explore possible outcomes and simulate how I believe this other mind will feel about it. When I do this, it is incredibly obvious that the person will be unhappy if I convert Earth into paper clips. When I do this, it is obvious why Alice asked, “Who is she?”

The intertwining of mentalizing and language is ubiquitous. Every conversation is built on the foundation of modeling the other minds you are conversing with—guessing what one means by what he said and guessing what should be said to maximize the chance the other knows what you mean.

The relationship between mentalizing and language can even be seen in the brain. Wernicke’s area, presumably the place where words are learned and stored, is right in the middle of the primate mentalizing regions. Indeed, the specific subarea of the left primate sensory cortex (called temporoparietal junction), which is highly selective for modeling other people’s intentions, knowledge, and beliefs, is entirely overlapping with Wernicke’s area—which is, as we have learned, required for people to understand speech

and produce meaningful speech.

Consistent with this, mentalizing skills and language skills in children are deeply interconnected. In preschoolers, there is a significant correlation between the development of language skills and performance in mentalizing tasks such

as false-belief tests. Disorders that impair mentalizing cause

similar impairments in language.

We are capable of puppeteering other minds because language is, it seems, built right on top of a direct window to our inner simulation. Hearing sentences directly and automatically triggers specific mental imagery. This is why if someone is saying something that upsets us, we cannot simply “not listen”; we must cover our ears, otherwise the words directly trigger simulations whether we like it or not.

But Wait . . . What About GPT-4?

In March 2023, OpenAI released GPT-4, its new and upgraded large language model. GPT-4 works in largely the same way as GPT-3—it predicts the next word based solely on the prior sequence of words. But unlike GPT-3, it is an even bigger model trained on even more data.

Amazingly, each question that I designed in this chapter to demonstrate a lack of common sense and physical intuition in GPT-3 was answered flawlessly by GPT-4. GPT-4 knew that if you looked upward in a basement, you would see the ceiling not the sky. GPT-4 knew that if someone threw a baseball one hundred feet above your head, you would not be able to catch it. GPT-4 could even answer theory-of-mind questions like the Sally-Ann Test.

How does GPT-4 do it? OpenAI did not add an inner world model or the ability to model the mind of others. Instead, in response to GPT-3’s frequent mistakes on commonsense and reasoning questions, OpenAI began training GPT-4 specifically on questions of commonsense and reasoning. They used a technique called “reinforcement learning from human feedback” in which humans punished GPT-4 when it answered questions incorrectly, and rewarded it when it answered correctly. They even pushed the GPT-4 to answer certain questions in specific ways to improve its performance. For example, OpenAI trained GPT-4 to think about commonsense questions by writing out each step, a trick called chain-of-thought prompting. Indeed, contrast how GPT-3 and GPT-4 each answer the bat and ball question:

Me: A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?

GPT-3:

GPT-4:

The ball costs $0.10 because if the bat costs $1.00 more than the ball, and the bat and the ball cost $1.10 in total, then the ball must cost $1.00 - $1.10 = $0.10 (incorrect)

Let’s denote the cost of the ball as x. Then, the cost of the bat would be x + $1.00, since it costs $1.00 more than the ball. The total cost of both items is $1.10. So, we can set up an equation: x (ball) + x + $1.00 (bat) = $1.10 Combining the terms for x, we get 2x + $1.00 = $1.10 Now, we’ll subtract $1.00 from both sides: 2x = $0.10 Finally, we’ll divide both sides by 2: x = $0.05 So, the ball costs $0.05 or 5 cents. (correct)

By training GPT-4 to not just predict the answer, but to predict the next step in reasoning about the answer, the model begins to exhibit emergent properties of thinking, without, in fact, thinking—at least not in the way that a human thinks by rendering a simulation of the world.

Even though GPT-4 correctly answers the simpler questions I outlined in this chapter, you can still find plenty of examples of GPT-4 failing on commonsense and theory-of-mind questions. GPT-4’s lack of a world model can be seen by probing deeper with more complicated questions. But it is becoming progressively more painstaking to find these examples. In some ways, this has become a game of Whac-A-Mole; everytime a skeptic publishes examples of commonsense questions that LLMs answer incorrectly, companies like OpenAI simply use these examples as training data for the next update of their LLMs, which thereby answer such questions correctly.

Indeed, the massive size of these models, along with the astronomical quantity of data on which they are trained, in some ways obscures the underlying differences between how LLMs think and how humans think. A calculator performs arithmetic better than any human, but still lacks the same understanding of math as a human. Even if LLMs correctly answer commonsense and theory-of-mind questions, it does not necessarily mean it reasons about these questions in the same way.

As Yann Lecun said, “the weak reasoning abilities of LLMs are partially compensated by their large associative memory capacity. They are a bit like students who have learned the material by rote but haven’t really built deep mental

models of the underlying reality.” Indeed, these LLMs, like a supercomputer, have a gargantuan memory capacity, having read more books and articles than a single human brain could consume in a thousand lifetimes. And so what seems like commonsense reasoning is really more like pattern matching, done over an astronomically enormous corpus of text.

But still, these LLMs are an incredible step forward. What is most amazing about the success of LLMs is how much they seemingly understand about the world despite being trained on nothing but language. LLMs can correctly reason about the physical world without ever having experienced that world. Like a military cryptanalyst decoding the meaning behind encryped secret messages, finding patterns and meanings in what was originally gibberish, these LLMs have been able to tease out aspects of a world they have never seen or heard, that they have never touched or experienced, by merely scanning the entire corpus of our uniquely human code for transferring thoughts.

It is possible, perhaps inevitable, that continuing to scale up these language models by providing them with more data will make them even better at answering commonsense and theory-of-mind questions.