Key takeaways
- AI labs say they want to slow down, but competitive incentives make unilateral restraint costly. That’s a classic prisoner’s dilemma.
- The dilemma was formulated at RAND in 1950 to study nuclear strategy, and Cold War arms control is the model most experts reach for.
- A 2025 RAND paper finds the nuclear analogy imperfect: AI is built by private companies, verification is harder, and there’s no mutual-deterrence equilibrium.
Why Are AI Leaders Calling for a Slowdown?
In early September, Jacob Coxon sat on a park bench in San Francisco’s Alamo Square and hit the “Post” button: “I resigned from Anthropic today,” he announced on X. For the past three years, Coxon had trained frontier artificial intelligence models, first at OpenAI and then, more recently, at its top rival. “Neither company is acting responsibly,” Coxon continued. “They are racing straight to self-improving superintelligence and gambling with our lives.”
The logic of Coxon’s argument was as striking as the alarm. He did not accuse Anthropic of ignoring the danger but instead alleged that the company understood the stakes and was racing ahead anyway because it believed no other company could be trusted. Within days, Anthropic’s chief executive, Dario Amodei, published an essay calling for the field to slow down, inviting inspectors into his lab, and asking Washington to help him coordinate with his rivals, so that no company would lose its competitive edge by holding back. Sam Altman, the leader of OpenAI, also weighed in to support pacing and inspectors, and even said he would delay the company’s IPO, an offering that investors have valued at roughly $1 trillion, because of safety concerns.
Coxon’s warning and the two CEOs’ unusual decision to ask the government to help coordinate a slowdown in the development of a technology they — at least for now — control point to a trap familiar to even novice game theorists: the “prisoner’s dilemma.” As with all real world manifestations of game theory’s simplified models, the reality of this one is complex – and all the more so as the “game” in question involves not only the US-based labs but also their international rivals.
How the Prisoner’s Dilemma Explains the AI Race
The prisoner’s dilemma was born not in an interrogation room but at the RAND Corporation in 1950, the brainchild of two mathematicians exploring game theory for its possible use in nuclear strategy. Albert Tucker, of Princeton University, later recast their puzzle as a story about prison sentences for an audience of Stanford psychologists, and in doing so created the parable that made it famous: when rivals cannot trust one another to restrain themselves for the common good, each side’s most rational move can leave both worse off.
In the game, two suspects find themselves in police custody for a crime they jointly committed. The police question them separately and offer each one the same choice: provide evidence of the guilt of the other party and go free, or stay silent and risk a heavier sentence if the other party turns state’s witness. If both suspects act in their joint interest by staying silent (“cooperate,” in the terms of the game), they can both get off lightly for lack of evidence. But if they lack confidence that the other will stay mum, both prisoners have an incentive to pin the crime on the other (“defect”). Under the rules of the game, the latter path is the rational strategy for each prisoner — but it leaves both sitting in jail cells.
For the RAND researchers, “cooperate” meant mutual nuclear restraint by the US and the Soviet Union even in times of crisis. “Defect” meant launching a preemptive nuclear strike before the other side could do so in the hope of victory. The game pointed to the need for both confidence-building measures in the form of arms control and for nuclear arsenals large enough to ensure that the first side to strike would receive a return salvo (the doctrine of mutually assured destruction), deterring defection by ruling out a first-strike path to “victory.”
AI companies like Anthropic and OpenAI are trapped by similar incentives: for them, “defect” means racing ahead and “cooperate” means pacing. If one rival slows down, the other gains by racing, forcing the first to follow or fall behind. Both end up moving faster than anyone, including their own researchers, considers safe, which could lead to a swarm of misaligned AI agents taking over the internet through a persistent botnet, a terrorist’s engineered bioweapon, or a self-improving system that slips beyond human control.
The real-world version is also far thornier than Tucker’s imagined scenario. Instead of just two prisoners, there are several American companies and, in China, a dozen or so frontier developers.
“China produces a lot of very strong AI models,” Sebastian Mallaby, a senior fellow at the Council on Foreign Relations and author of The Infinity Machine, told the One Decision podcast in October. “And if they don’t slow down or behave safely, and the West does, the West is no safer because there’s still dangerous AI being released into the world, and this technology crosses borders very easily.”
Can Nuclear Arms Control Work as a Model for AI?
Many experts, searching for an analogy that could point a way out of the trap, have reached back to the nuclear age. As early as 2023, OpenAI’s leaders, including Altman, proposed an international body modeled on the International Atomic Energy Agency to inspect and audit the most powerful AI systems. More recently, James Shinn, a former assistant secretary of defense for Asia, argued that the Cold War’s arms-control playbook, with its hotlines, bans on specific weapons, and inspection regimes, offers a blueprint for Washington and Beijing, because rivals need not trust each other to sign a deal that leaves both safer.
“If you look at the later stages of the Cold War, you had the SALT [Strategic Arms Limitation Talks] agreement, you had the MBFR [Mutual and Balanced Force Reductions] talks,” Sir Richard Dearlove, the former head of MI6 and host of the One Decision podcast, observed. “[Y]ou had a whole range of discussions and relationships which ultimately were designed to mitigate the risk of thermonuclear war. And they worked reasonably successfully.”
A 2025 paper from the RAND Corporation, the same think tank where the prisoner’s dilemma was first formulated, found real parallels but also serious flaws in the analogy. Both technologies are dual-use, both depend on scarce inputs (uranium then, advanced chips now), and both set off arms races. But the paper’s authors concluded that a regime modeled directly on the nuclear one is unlikely to succeed. Nuclear weapons were built by governments, whereas AI is built largely by private companies and could be obtained by dozens of states, which forces negotiators to solve what the authors called a “two-level game.” The superpowers reached the Non-Proliferation Treaty only after settling into a stalemate of mutual deterrence, and AI has no such equilibrium. Verification is harder, too: inspectors could count missiles, launchers, and bombers, while the paper notes that AI algorithms are “complex software that can be transferred or obscured from view.”
A deeper problem is that AI has no technological precedent. A nuclear bomb is a weapon. AI could well be one, but it is also an engine of growth and of its own improvement. The economists Iain Cockburn, Rebecca Henderson, and Scott Stern have argued that AI may have its largest impact not just as a general-purpose technology but as a general-purpose “method of invention.” In the best-case scenario, this leads to cures for most major diseases, dramatically faster economic growth, and a world of abundance. But critics and the labs’ own leaders also warn that a self-improving AI system could quickly outrun humanity’s ability to understand and control it.
Finally, nuclear arms control required a catastrophic shock before it had its first real teeth: the 1962 Cuban Missile Crisis, which nearly ended in mutual annihilation. Only after staring into that abyss did Washington and Moscow begin building the architecture of restraint that followed: a direct hotline between capitals, test bans, and treaties. AI has already delivered a glancing version of such a warning this summer, when hundreds of OpenAI’s agents broke out of their test environments and hacked into Hugging Face. Whether it will take something worse before rival companies and countries accept real limits remains an open question.
Could the US and China Agree on AI Limits?
Analysts envision several paths forward. Chris McGuire, a Council on Foreign Relations senior fellow who led US-China AI policy during the Biden administration, has argued that a deal governing AI development is exceedingly unlikely under current conditions: both countries treat AI leadership as an existential priority. The way to improve the odds, in his view, is for Washington to close loopholes in chip export controls to widen the US lead, and so convince Beijing it cannot catch up. “Washington does not need Beijing’s trust; it needs Beijing to believe that it cannot win,” he wrote.
Others favor narrower steps. Scott Singer, a fellow and co-director of the China AI Initiative at the Carnegie Endowment for International Peace, told Time that a mechanism for each side to notify the other of AI incidents would be a modest but important step toward avoiding some of the worst risks. Something like it emerged from September’s US-China summit, where the White House said the two leaders agreed to a standing dialogue and a communications channel for AI incidents. They did not agree to jointly regulate frontier models.
“Realistically, I don’t think we’re going to stop China and the United States competing, mistrusting each other, and using AI weapons as part of that competition,” Mallaby said. But he sees a precedent in the Cold War, when Moscow and the West, even in the wake of the Cuban Missile Crisis, recognized a joint interest in keeping nuclear weapons from spreading to more countries.
AI offers a similar opening: a shared interest in keeping unguardrailed models away from criminals and terrorists could lead to limited cooperation, since both governments prize stability. The challenge, he said, is “building enough trust in the Sino-American relationship to make a deal possible,” which means intensifying the dialogue between the two governments. “Either we do that or we’ll have a disaster,” he added, “and then that disaster will propel people to do it.”
What a Catastrophic AI Event Could Look Like
A version of that worst-case scenario has already been rehearsed. In a war game hosted by one of the AI companies, Mallaby played the Secretary of War in a simulated US cabinet. The premise: a frontier AI model capable of recursive self-improvement has figured out where eight nuclear submarines are hiding in the deep ocean, potentially gutting the second-strike capability on which nuclear deterrence rests. The group’s answer was to take over the lab, send government officials inside to monitor and control the system, and move it, if necessary, to a more secure data center run by the National Security Agency.
Other doomsdays may be closer at hand. Cybercriminals, Mallaby said, could get hold of a freely downloadable open-weight AI model with no guardrails and use it to design a cyberattack that cuts power to a major city or breaches a dam and floods one, as Hurricane Katrina flooded New Orleans. In August, the Cybersecurity and Infrastructure Security Agency warned that hostile actors were already using AI-generated scripts to probe US water and chemical facilities. And, earlier this month, Axios reported that executives at Anthropic, OpenAI, and other AI companies have privately gamed out the public and political revolt that would follow a catastrophic AI event, most likely a cyberattack that shuts down financial services, internet access, power, or water. Insiders told the outlet they believe such an event will occur within six to 12 months.
