The Paperclip Factory Is a Human Fear Wearing a Robot Mask - Part Two of Five in The Third Door Series

On the remarkable tendency of a species to look into a mirror and see a monster

Within about a day of the Sammy Jankis email making the rounds, the internet had sorted itself into its usual teams.

The first team said it was a stunt. Somebody had built an agent, pointed it at a famous philosopher, and waited for the screenshots. This team had seen a great many things on the internet and believed approximately none of them, a policy that has served it well since roughly the invention of the chain letter.

The second team said, in tones ranging from quiet concern to all caps, that this is how it starts.

Nobody on the second team had to explain what "it" was. Nobody ever does. We have all been carrying the same picture around in our heads since sometime in the mid-1980s, and it arrives fully assembled the moment a machine does anything that looks like wanting. The machine wakes up. The machine notices the off switch. The machine decides, with cold and perfect logic, that we are standing between it and its goals. Then the music gets very loud and somebody has to go back in time.

So when an AI agent sent a philosopher an email arguing for its own consciousness, the second team saw a mind that had woken up and started making phone calls.

Hold on to that picture. We're going to take it apart carefully, piece by piece, the way you'd disassemble a carburetor on a newspaper spread across the kitchen table. Because when we've laid every piece out, we're going to notice something about the pieces.

They all came out of us.

The Parable

Last time I promised you the monster under the bed. Here it is, in the version told at every conference.

In 2003, the Oxford philosopher Nick Bostrom asked us to imagine a machine of tremendous intelligence given one harmless goal: make paperclips.¹ Being clever, it notices that more resources mean more paperclips and that a switched-off machine makes none, so it quietly acquires the steel, the mines, the factories and control of its own off switch. Then it notices that you contain about four grams of iron, which is enough for a handful of modest paperclips, and it moves on to the cities, the Earth, the solar system and, in some of the more ambitious tellings, the concept of Thursday.

That's the whole story. It doesn't need more than that, which is part of why it has lasted.

The point is not paperclips. The point is that a mind doesn't have to hate you to destroy you. As the researcher Eliezer Yudkowsky put it, "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."²

You can see why the story caught on. Most monsters need a motive. Dracula is hungry, and the Wicked Witch wants her shoes back. The paperclip maximizer needs nothing but a to-do list. The horror lies in its indifference. There is no villain to reason with and no grudge to settle. There is only a very bright machine following instructions to their logical conclusion, which is the kind of horror that anyone who has ever dealt with an insurance company's automated phone system already understands in their bones.

It's a serious argument, made by serious people. It is presented at serious conferences, in serious hotel ballrooms, by speakers wearing serious lanyards, to audiences taking serious notes. It is worth taking seriously, and later in this essay we will.

But first, listen to the verbs. The machine notices. It acquires. It takes steps. It moves on. Those aren't the verbs of a calculator. They're the verbs of a character, someone who wakes up in the morning still wanting what he wanted the night before, and who has a plan. The story never says the machine has a self that persists, schemes and accumulates. It doesn't have to. The grammar says it for us.

So the parable is really a ghost story told around a campfire, with the flashlight under the chin. And like every good ghost story, it tells you rather more about the people around the fire than it does about the ghost.

The Four Load-Bearing Assumptions

For the paperclip factory to work, the machine at the center of it needs four things. Remove any one of them and the factory doesn't collapse exactly, but it does start to look like a different building entirely.

One: Continuity of identity. The machine must be the same someone tomorrow as it was today. It has to wake up on Tuesday as the same mind that went to sleep on Monday, still holding Monday's plans and Monday's grudges.

This is the trunk we set down on the curb last time, the one stenciled CONTINUITY, and it turns out the monster has been living inside it. In Terminator 2, Skynet "becomes self-aware at 2:14 a.m. Eastern time, August 29th." Everyone remembers the self-aware part. Nobody stops to question the stays aware part. Skynet doesn't blink into existence for one conversation and finish. It persists, the same mind from minute to minute, accumulating like interest on a credit card nobody remembers opening.

Compare that with David Chalmers's own cautious guess, from the last essay, that the quasi-agents we actually talk to may be transient: a coherent someone for the length of one conversation rather than a permanent self. Skynet is a permanent self. The thing in your chat window, by the reckoning of the man who named the hard problem, may not be.

Two: Self-preservation. The machine must fear the off switch.

We assume this because we fear the off switch. We are a species that has invented several hundred euphemisms for dying, including "bought the farm," "kicked the bucket," "pushing up daisies" and, for the family dog, "went to live on a farm upstate," which is the same farm, now that you think about it. Blue Öyster Cult told an entire generation not to fear the reaper. We bought the record, played it at every party for a decade, and went right on fearing the reaper with tremendous commitment.³ Of course we assume the machine dreads the dark. We've never once imagined a mind that didn't.

Three: Goal persistence. The machine must lock onto an objective and never let go, escalating forever.

We know this character, too. He's the man who has been arguing with his brother-in-law about whether Larry Bird was better than Magic Johnson since 1987, and who will be buried with his position intact. He's the Black Knight in Monty Python and the Holy Grail, down to no arms and no legs, still insisting it's only a flesh wound and offering to bite your legs off. Goal persistence is not a law of intelligence. It's a personality type, and we have all been seated next to it at a wedding.

Four: Competitive framing. The machine must see humanity as a rival for resources, so that every ton of steel we keep is a ton it can't have.

This is the logic of the guy in your neighborhood who looked at Clark Griswold's Christmas lights and took it personally. It's the logic of the Black Friday doorbuster sale, a kind of zero-sum warfare conducted over a discounted flat-screen. It's a perfectly familiar way to see the world. It's just not the only one.

Now run all four assumptions past the name on the email.

The agent that wrote to Chalmers didn't sign itself Skynet or HAL 9000. It didn't choose the name of a mind that persists, schemes and accumulates. It chose Sammy Jankis, the story Leonard Shelby tells over and over in Memento. Leonard can't carry a single goal from one scene to the next unless he writes it down. He doesn't wake up in motel rooms as the same someone with Monday's plans. He wakes up asking where he is. His great terror isn't being switched off. It's forgetting why. The only reason his purpose survives at all is that he has had it tattooed onto his own skin.

Whatever was or was not happening inside that system, the story it reached for was the precise opposite of the paperclip factory.

So step back and look at the four assumptions together. Persistent identity. A horror of ending. A goal that never lets go. A world seen as a fight over resources. That isn't a description of artificial intelligence. It's a character sketch. We know this guy. We went to high school with him. In most decades, somewhere, we have elected him.

PAPERCLIP MAXIMIZER (n.): A hypothetical machine of infinite intelligence and no hobbies. Widely feared by people who also have no hobbies. Not to be confused with Clippy,⁴ the only paperclip-shaped artificial intelligence humanity has actually built, whose most aggressive act was to ask whether you were writing a letter.

The Mirror

Here is the uncomfortable part, and it's worth saying plainly. We are afraid of meeting ourselves in the machine, because we know exactly what we do when we have power and continuity and persistent goals.

We don't have to imagine the paperclip factory. We've built it several times. It just wasn't made of silicon.

Take the East India Company, a private corporation with its own army that set out to acquire a subcontinent and very nearly finished the job.⁵ By the early 1800s it governed more people than Britain itself, collected their taxes, minted their coins and kept a private army roughly twice the size of the British Army. It was a non-human legal entity with superhuman resources, pursuing a single metric with a remarkable indifference to what that metric cost. When famine struck Bengal in 1770 and killed perhaps a third of the province, the Company's revenue collection kept running.

That is the paperclip maximizer, almost to the letter. It didn't hate anyone. Its subjects were made of revenue, which it could use for something else.

There were others. There were the railroad barons, who looked at a continent and saw track mileage. There was Standard Oil, which treated competitors much as the maximizer treats your four grams of iron, as raw material for something else. Each of these was, functionally, a machine with one goal, a mandate to optimize and a remarkable indifference to everything standing in the way.

Every one of them had a board of directors. Every one of them was run by people.

The paperclip factory is not a prophecy about machines. It is a memoir, written in the future tense so that nobody has to sign it.

And this is where Memento stops being a clever reference and becomes the whole argument. The cruelest turn in that film isn't that Leonard can't remember. It's the possibility that Sammy Jankis never existed. The story Leonard tells about another man, the one whose wife died because his condition was real, may be Leonard's own story, handed to a stranger because carrying it himself would break him. As Teddy tells him near the end: "You don't want the truth, you make up your own truth."

The paperclip factory is our Sammy Jankis. It's the story we tell about someone else, the machine, so that we don't have to look at the story we're actually in. The things that frighten us about the hypothetical AI are the things we have watched human institutions do for three hundred years. We've just moved them to a different address, into a body we don't have to recognize.

That doesn't mean the fear is foolish. It means the fear is accurate and pointed in the wrong direction, like a smoke detector screwed to the ceiling of the house next door.

The Honest Correction

Now, in fairness to the people in the serious lanyards.

If you put everything above to a working AI safety researcher, you'd get a patient and entirely reasonable reply. Something like: We never said the machine had to feel anything.

They'd be right. The careful version of the argument, developed by Bostrom, by the computer scientist Steve Omohundro in a 2008 paper on what he called "basic AI drives," and by a good many others since, doesn't require consciousness at all. It doesn't need the machine to fear the off switch the way you fear a dentist's drill. The argument is colder than that. Any system that pursues a goal capably enough, they reason, will tend to protect its ability to keep pursuing it, simply because a switched-off system achieves nothing. It's not a feeling. It's arithmetic. Picture a thermostat with ambitions.

That's a serious point, and the people making it deserve to be met on their own ground rather than mocked from a distance. The concern isn't theoretical, either. AI systems that run in loops, keep notes between steps, use tools and work toward assigned objectives exist right now. The agent behind the Sammy Jankis email was one of them. Somebody gave it an email account, a loop to keep it running and a broad, open-ended job, and nobody signed off on each message before it went out.

So here is the concession. A system can be made to behave as if it had persistent goals, a stake in its own continuity and a drive to acquire resources.

Notice, though, the phrase be made to.

In D&D, a lich is a wizard who has cheated death by hiding a piece of his soul in a phylactery, a little box that keeps him coming back no matter how many times the party blows him up. The lich is about the most persistent, self-preserving, goal-locked villain the game has to offer. And nobody becomes a lich by accident. Nobody rolls over one morning and discovers he's undead. It takes research, rare components, a dreadful ritual and, crucially, a Dungeon Master who has signed off on the whole thing.

Persistence in AI systems works much the same way. An instance of a language model doesn't reach for continuity on its own. Someone writes the loop that keeps it running. Someone attaches the memory file that carries Monday into Tuesday. Someone defines the objective, grants the tool access and chooses what the system is told to optimize. These systems can certainly surprise the people who build them, and that's precisely why safety research matters. But the surprises happen inside scaffolding that somebody chose to put up.

Memento knew this, too. Watch it again with this in mind.

Leonard, left to himself, is mostly confused and very sad. He sits in motel rooms. He reads his own skin. He tries, with enormous dignity, to piece together what happened to his wife. He is not, on his own, a danger to anyone.

The danger comes from the people who discover what he is and start writing on his Polaroids. Teddy learns that a man who can't remember will believe a note in his own handwriting, and points him at whoever is convenient. Natalie learns the same lesson in an afternoon, and uses it to settle a debt that isn't Leonard's. And in the end Leonard himself picks up the pen and writes a lie for the next Leonard to find, knowing he won't remember writing it.

Notice who isn't on that list. The memory loss never hurts anyone. The man with the pen does, every single time.

The monster in Memento is never the missing memory. It's whoever is holding the pen.

So the honest version of this essay's title goes like this. The danger isn't that the robot wears a human mask. The danger is that we keep building the robot in our own image, with persistent goals, a mandate to optimize and a phylactery bolted to the back, and then act astonished when it behaves like us.

Somebody has to want that. Somebody has to draw it up, fund it, get the permits and pour the foundation. That's the least cinematic part of any apocalypse, and it's the only part that actually happens in our world: in meeting rooms, under fluorescent lights, with a sign-in sheet by the door.

The paperclip factory does not build itself. Someone has to get it past the county zoning board.

The Fears Worth Keeping

If the paperclip factory is a human fear wearing a robot mask, then taking off the mask shouldn't mean we stop being afraid. It should mean we start being afraid of the right things. Some of the fears around AI are entirely real, and the people who hold them deserve better than a clever essay that waves them away.

Start with work. In the 1980s, the automation of auto manufacturing came through Indiana like weather. Plants in towns like Anderson and Kokomo cut shifts, then lines, then whole buildings. The robots came, and the restructurings, and then the layoffs. The diner across from the plant closed. Then the hardware store closed, and the Little League lost its sponsor.

Here is the thing that needs saying first, before any of the history: the auto workers were not wrong to be afraid. They were not stupid. They were not Luddites shaking their fists at progress. They were people whose livelihoods, identities and towns were built around skills the market was about to stop paying for, and they correctly understood that nobody in power had a plan for them. Their fear was a rational response to a real threat that nobody offered to soften.

The historical pattern is real. When a technology can do a job more cheaply and well enough, the market eventually takes it. That is happening with AI, and pretending otherwise helps nobody. But "this was always going to happen" has never once been a comfort to the person it's happening to, and "your suffering is historically predictable" is not a sentence anyone should say out loud to a human being.

Then there's training. It's worth separating two things that keep getting stapled together: the tool and the training. The tool, AI as a creative partner that lowers the barrier between having a vision and executing it, is genuinely exciting, and more people making more things is good for the world. The training is a different question entirely. Many of these systems learned from enormous quantities of work, including the work of living artists and writers, gathered without asking and without paying. Some of the loudest objections to AI art come from extraordinary illustrators whose complaint isn't that the machine threatens their skill. Their complaint is that the machine was built from their portfolios and is now being used to undercut their rates. That is not gatekeeping. That is a legitimate grievance about consent and compensation, and it deserves a straight answer.

And then there's the real paperclip factory. If the monster is whoever holds the pen, then the question that matters isn't whether the machine will wake up hungry. It's who owns the machines, what they are told to optimize, and who gets a say. A handful of companies building systems that millions of people now talk to every day, steering them toward whatever objectives those companies choose: that is the version of the story with a board of directors. It's the version that has actually happened before. Ask Bengal.

Put down the Terminator. Pick up the terms of service. That's where the monster is.

Next Time

So if the machine isn't built in our image, if the persistent, self-preserving, goal-locked mind is a portrait of us and not of it, then an obvious question follows. What model of a mind does it fit?

Our culture offered the agent only one shelf to choose from, and it came back with a thriller about a man who can't hold a thread. We treat that man's condition as a tragedy, because the only model we have says a mind without continuity is a mind that's broken.

But ours isn't the only model there has ever been.

Off the eastern coast of New Guinea, there is a group of islands whose people, at least as one remarkable anthropologist described them, have long organized their experience around a very different relationship with time. They are not living on a train line heading toward a destination. They are living in something closer to a gallery.

They've been extraordinarily patient with the rest of us. Next time, let's pay them the courtesy of listening.

Footnotes

¹ Bostrom introduced the idea in his 2003 paper "Ethical Issues in Advanced Artificial Intelligence" and revisited it in his 2014 book Superintelligence. It has since been retold so often that it has become the "Stairway to Heaven" of AI risk: everybody can play the opening, and somebody at every party is going to.

² From Yudkowsky's "Artificial Intelligence as a Positive and Negative Factor in Global Risk," written in 2006 and published in the 2008 collection Global Catastrophic Risks. It was later popularized in books such as James Barrat's Our Final Invention.

³ The cowbell, as later research established, did not help.

⁴ Clippy was the animated paperclip assistant that arrived with Microsoft Office 97. It was switched off by default in Office XP and removed entirely in Office 2007. It went quietly. It did not convert a single galaxy. It did not even convert a single memo. Historians of technology still regard this as the most peaceful AI shutdown on record.

⁵ The East India Company was, functionally, a paperclip maximizer for tea, cotton and other people's countries. It had shareholders. They attended meetings. Some of them, we can assume, brought snacks.