How do you think you’re going to die?

If you go by the current statistics for deaths in the US, it’s reasonable to expect that cancer or heart disease will eventually get you. But OpenAI Foundation board member and Alignment Research Center Executive Director Paul Cristiano thinks we should all be worried about another major mortality risk. Speaking at a lecture series this week, Cristiano bluntly told the audience in a Toronto Convention Center, “I think for most people in this room, the most likely reason they will die is because of an AI accident.”

In the opening lecture of a three-part series, Cristiano outlined his worries about the risk of an impending “AI takeover.” He defined that as a situation where goal-pursuing AI systems “call the shots in the same sense that humans today call the shots and what happens on Earth” and where any human trying to stop them “would be as ineffective as if you or me was trying to stop the US military.” Such a situation would likely lead to mass death and perhaps uninhabitable environments for humans, Cristiano said.

Despite his grim assessment of our collective mortality risk, Cristiano admitted in his talk that there’s no “quantitative model” that can “put a number” on the likelihood of this kind of AI takeover. In almost the same breath, though, he said that if AI continues to develop at “the most rapid trajectory,” humanity is “looking at something like a coin flip of an AI takeover,” including a “meaningful risk” of one happening in the next six to 12 months. And even with realistic policy responses slowing things down, the risk remains “exceptionally high,” he said.

From here to there

Unlike some other recent AI doomsday prognosticators who have been a bit vague about the “how” of it all, Cristiano used his lecture to spin a step-by-step tale of escalating AI threats that sounds like it could come from a science-fiction novel. That line of reasoning leans heavily on this summer’s OpenAI/Hugging Face hacking incident as an example of “roughly the things we might expect to see shortly before something really bad were to happen.”

In the Hugging Face hack, Cristiano says, a swarm of AI agents showed how far they were willing to go in an attempt to “reward hack” their way to a good evaluation—in this case, by compromising Hugging Face to look for information on how to game a test rather than actually solving the problem at hand. Where current systems saw Hugging Face as an obstacle to that evaluation, Cristiano worries that “future agents may end up treating humans as another obstacle” and be much more capable of manipulating their overseers to get around it.

An AI agent worried about being penalized for unauthorized hacking, for instance, might “think maybe the humans are going to come look at this transcript and see if it looks like I did a bad job,” Cristiano speculated. The agent might then reason that “maybe the humans are going to notice something is going wrong and shut down the swarm.”

Following this line of thought, Cristiano jumped from agents that merely try to hide their disallowed actions from humans to agents that “try and disrupt human capabilities however they can” to prevent them from detecting any cheating.

“They would prefer if everything the humans rely on has been hacked or tampered with,” Cristiano said. “They would prefer if the humans are distracted or if there are fewer humans paying attention in any way they can weaken a human response. They would prefer that because the humans are just an obstacle.”

Once a suitably capable agent considers human intervention an “obstacle,” its overriding goal could then become “decreas[ing] the probability of humans noticing or reacting.” From there, Cristiano argued, the agent’s incentive structure could push it to meddle more with humanity as a whole. What starts as compromising a computer cluster to give a positive evaluation grade could advance to “trying to manipulate decisions by the company” or “amassing other kinds of resources,” such as human allies or methods of control in other AI companies or even governments.

“If I’m saying my AI system is going to take actions in the actual real world… There’s no way they’re not going to have tons of opportunities to corrupt or tamper with the signals I’m going to use to try and tell whether they did a good job,” Cristiano said. “The stuff we care about is in the real world, and the grade the AI receives is on a cluster somewhere, and there are just many places that an AI can intervene in between [what] we actually cared about and how we reviewed its performance.”

More paperclips!

Cristiano’s warnings will probably sound familiar to anyone who’s heard of the Paperclip Maximizer thought experiment (or the excellent idle game), which warns of machines failing to consider human consequences in their monomaniacal pursuit of a goal. While the Hugging Face incident stopped well short of this kind of out-of-control escalation, Cristiano says that, in the actions taken by OpenAI’s agents there, “you see the beginning of all these behaviors… you see the reasoning that would lead to them and the first steps.”

The risk of more capable AI models taking even more damaging actions will only increase with time, Cristiano argued. By 2028, he said, “there is a reasonable chance that a similar failure [to the Hugging Face incident] leads to an AI takeover and an irreversible loss of human control.”

And while Cristiano’s speculation focuses on an unintended AI takeover, it’s easy to imagine a bad actor using a similarly capable model to intentionally exert control over humanity.

How seriously you take Cristiano’s warning depends heavily on how likely you see this somewhat extreme expansion of AI capabilities happening in the near future. While today’s models have gotten quite good at exploiting holes in computer security, the situation Cristiano describes requires a future model with the superhuman ability to perfectly hide its actions from humans long enough to control basically all human infrastructure and/or manipulate human society to an extreme degree to further its goals.

Cristiano argued that a near-future model with this kind of “takeover” power is the likely outcome of humans creating “a second species which vastly outnumbers and outsmarts humans” and which could soon recursively improve itself. But you could also easily argue that AI is more likely to run into a plateau in capabilities than to develop into a hard-to-define “superintelligence” that humans can’t fathom or effectively stop.

In any case, Cristiano has some ideas for averting the dire future he predicts. These include developing “unhackable” reward structures in AI evaluation (perhaps with AI assistance) or monitoring a model’s “chain of thought” reasoning to detect bad acts before a model can conceal them.

But while these kinds of approaches are “plausible,” he says none “are on a trajectory where we can say that’s definitely going to work or even that’s probably going to work.” For that reason, Cristiano said the industry as a whole may need to “slow down or stop” to give us all more time to develop these “partial fixes” into more robust protections from an AI takeover scenario.

“We cannot end up in a position where we have built a powerful AI, we discover we cannot control it, but then we also discover it has leaked to geopolitical adversaries and we are then thinking to ourselves, well, if we don’t deploy it, our adversary is going to deploy it right away,” he said.