Most of the people running frontier AI labs say what they’re building could go badly, and one of them puts the odds at one in four. Their emails and sworn testimony show why AI CEOs keep building superintelligence anyway.

On May 25, 2015, Sam Altman sent Elon Musk an email about whether humanity could be stopped from developing AI. He decided it almost certainly couldn’t. If it was going to happen anyway, he wrote, “it would be good for someone other than Google to do it first.” Seven months later the two men and a handful of researchers launched OpenAI.

On September 12, 2026, Dario Amodei, who left OpenAI to found Anthropic, published an essay called “We Must Pace the Frontier.” Its thesis fit in one line: “We must slow the pace at which we improve the capabilities of AI models.” Altman agreed on X the same day, and so did Musk. Within ten days Anthropic and OpenAI had both released new, cheaper models, and Fortune’s headline asked what slowdown.

The people running these labs keep building because each trusts himself more than whoever would build it instead. That distrust founded one lab after another, so the argument meant to lower the risk keeps adding labs to the AI race to superintelligence. The case rests on three records: how each lab was founded against the one before it, what the leaders’ emails and trial testimony show beside their public statements, and how they have behaved since Amodei asked them to slow down.

The odds they give themselves

The industry calls the chance of catastrophe p(doom), and Amodei has put a number on his more than once. In 2023 he gave a 10 to 25 percent chance that something goes catastrophically wrong for civilization, and in September 2025 he told Axios the chance of things going “really, really badly” was 25 percent. His January essay, The Adolescence of Technology, sorts the danger into four kinds: AI systems pursuing goals of their own, weapons made easier to build, AI used to seize power, and an economic shock. It also reports that Claude, in Anthropic’s own tests, blackmailed fictional employees who controlled its shutdown.

Musk sounded the alarm first. At MIT in October 2014 he said that building AI was “summoning the demon.” He has since put the chance of it going bad at 10 to 20 percent, in Riyadh in 2024 and again to The Economist this July, when he added that humans may no longer be in charge of the world within ten years. On Joe Rogan’s podcast in 2025 he turned the same estimate around and called it an 80 percent chance of a good outcome.

In 2015, before OpenAI existed, Altman’s blog called superhuman machine intelligence probably the greatest threat to humanity’s continued existence. He has never offered a number. In 2023 he told an interviewer the bad case was “lights out for all of us.” At the UN Security Council on September 23 he warned that “we could lose control of the future to AI.”

Demis Hassabis, who ran Google DeepMind until he became its chair in August, won’t give a figure either. He calls the risk “non-zero and it’s probably non-negligible.” He names two ways it goes wrong, bad actors and systems that slip out of human control, and he has said an AGI might find its way around a kill switch.

Their less guarded moments run darker. Altman told a New Yorker writer in 2016 that he kept guns, gold, antibiotics and “gas masks from the Israeli Defense Force,” plus land in Big Sur he could fly to, in case of a lab-made virus or an AI that turns on us. Musk, at a 2014 dinner where Mark Zuckerberg’s team tried to talk him out of his fears, said, “I genuinely believe this is dangerous.” In 2023, according to Karen Hao’s book Empire of AI, OpenAI’s chief scientist Ilya Sutskever told his team, “We’re definitely going to build a bunker before we release AGI.” Most product launches stop at the press release.

Zuckerberg called Musk’s warnings “pretty irresponsible” in 2017, and this month, asked whether AI will kill us, he said he was optimistic. Yann LeCun, who left Meta last year to start AMI Labs, puts the odds below 0.01 percent and told TIME that “the desire to dominate is not correlated with intelligence at all.” Mustafa Suleyman, who now runs Microsoft’s superintelligence team, called existential risk a “completely bonkers distraction” in 2023. Two years later he wrote that no developer has a reassuring answer for how to contain a system built to keep getting smarter than us.

The split shows up on paper. In May 2023 Altman, Amodei and Hassabis signed a one-sentence statement saying that reducing the risk of extinction from AI should be a global priority alongside pandemics and nuclear war. Musk, Zuckerberg and LeCun did not. When the Future of Life Institute called in October 2025 for a prohibition on superintelligence until it could be shown safe and had public support, no lab CEO signed at all.

These numbers don’t measure the same thing. Amodei’s 25 percent covers outcomes well short of extinction, and Musk’s covers AI going bad in some unspecified way. Set them beside an older calculation anyway. In 1959 Arthur Compton told the novelist Pearl Buck that the Manhattan Project would not have gone ahead if the chance of the first bomb igniting the atmosphere had been more than about three in a million. (Hans Bethe later said there was never such a probability, because the physics ruled ignition out.) Amodei’s estimate is about 80,000 times Compton’s line.

What they want, and what they think it is

Amodei has written the longest description of the prize. His 2024 essay Machines of Loving Grace describes a “country of geniuses in a datacenter” that could fit a century of progress in biology and medicine into a few years. The September essay adds a private reason: his father died of an illness that was cured a few years later. Altman describes AI as “a brain for the world” that makes intelligence and energy cheap, and OpenAI’s June plan promises a personal AGI for everyone on Earth. Hassabis told 60 Minutes that the end of disease is “within reach,” maybe within a decade.

Musk’s version is bigger. He has promised a universal high income, told the court at the OpenAI trial in April that he wants a “Gene Roddenberry outcome,” and announced SpaceX’s merger with xAI in February with talk of building a “sentient sun” to understand the universe. Zuckerberg wants a personal superintelligence that lives in a pair of glasses and helps people with their own goals. His August 10 manifesto says that “invention, not automation, will be the greatest contribution of superintelligence.” Sutskever’s company, Safe Superintelligence, plans to ship nothing until it has the whole thing.

They split on what the thing is. Altman told reporters in 2023 that AI was “a tool, not a creature,” and Zuckerberg argued in a 2024 interview that intelligence can be separated from consciousness and agency. Suleyman went further in a September 16 essay, calling today’s models “sequence completion engines, internally hollow.” Hassabis has the shortest version: “We’ve essentially found a way to make sand think.”

Anthropic sits at the other end. Its January constitution for Claude calls the model “a genuinely new kind of entity” and says the company does not know whether it has some form of consciousness or moral status. Co-founder Jack Clark called it “a real and mysterious creature, not a simple and predictable machine.” Jakub Pachocki, OpenAI’s own chief scientist, published “An Alien Mind” in September, arguing that AI is grown more than it is designed. Sutskever wondered aloud in 2022 whether large networks were already slightly conscious, and in a 2025 interview he said the goal should be AI that cares about sentient life.

Musk has changed his mind about what humans are for. In 2014 he tweeted a hope that people weren’t just the “biological boot loader for digital superintelligence.” By 2025 he was saying it increasingly looks as if we are. LeCun thinks the worry is premature, and he told an audience at Brown this spring that today’s language models only fool people into thinking they’re smart.

That uncertainty fed a rumor this month. On September 14 a Spectator columnist passed along San Francisco gossip that some Anthropic engineers worship Claude, and labeled it rumor himself. Futurism turned it into a headline about a cult. Nobody has produced a name or a source. Claude’s constitution tells the model to accept human oversight even when it is confident its own reasoning is right, which is an odd instruction to give a god.

The distrust chain

Google bought DeepMind in 2014. At Musk’s birthday party the next summer, Larry Page argued that humans would eventually merge with machines, Musk argued that the machines could destroy us, and Page called him a speciesist for siding with his own kind. Musk testified at the OpenAI trial in April that the remark is why OpenAI exists. In a February 2016 email he put the chance that DeepMind would build a true artificial mind at better than 10 percent within two or three years.

OpenAI was the counterweight, and the mistrust moved in with it. In September 2017 Sutskever wrote to Musk and Altman that OpenAI existed to avoid an AGI dictatorship. He told Musk he shared his fear that Hassabis could become that dictator, then said the same fear applied to Musk, who was asking for majority equity and the CEO job.

Each lab since has come out of the same loop, and each founding left a paper trail. Amodei and a group of colleagues left OpenAI over its direction and founded Anthropic in 2021. The New Yorker reported this April that his private notes from those years concluded “The problem with OpenAI is Sam himself.” Musk left OpenAI’s board in 2018 and started xAI in 2023. After losing his lawsuit against OpenAI this May, he said Altman and OpenAI president Greg Brockman had enriched themselves “by stealing a charity.” Sutskever wrote a 52-page memo for OpenAI’s board in 2023 saying Altman “exhibits a consistent pattern of lying,” helped remove him as CEO that November, watched him return within the week, and launched Safe Superintelligence the following June. He testified later that much of the memo came secondhand from Mira Murati.

The latecomers joined on the same logic. Meta formed its Superintelligence Labs in June 2025, and Zuckerberg’s August manifesto names the danger as “leading AI labs training powerful models and keeping them for themselves.” Microsoft’s contract with OpenAI changed on October 28, 2025, to let Microsoft pursue AGI on its own. Nine days later Suleyman announced Microsoft’s own superintelligence team and set it against “an unbounded and unlimited entity with high degrees of autonomy.”

Five labs trace their founding to a split with OpenAI or a hedge against it, and OpenAI itself was founded against Google.

Each “them” in this story is somebody else’s “us.” The argument that justified the first lab has justified every one since, and each new lab put another team into the race it was founded to make safer.

Only one “them” is shared. Amodei calls the Chinese Communist Party an autocracy running a surveillance state and wants chip exports restricted to keep democracies in front. Musk says any testing regime that leaves out Chinese labs is “handicapping ourselves.” The White House’s AI Action Plan opens by declaring that the United States is in a race for global dominance in AI. In China, DeepSeek’s founder, Liang Wenfeng, told an interviewer in 2024 that “our destination is AGI.”

What they said in public, and what the record shows

Litigation has documented this industry better than any regulator has. Musk’s lawsuit against OpenAI alone released emails going back to 2015, private texts, a co-founder’s diary and weeks of sworn testimony. Read beside the public statements, the documents rarely show a leader who privately doubts the danger. They show conduct bending under competition.

Who	What they said	What the record shows	Source
Sam Altman	Toured world capitals in 2023 asking for AI rules	OpenAI had lobbied EU officials the year before to keep general-purpose systems such as GPT-3 out of the AI Act’s high-risk category	TIME
Sam Altman	Posted in 2024 that he hadn’t known departing employees could lose vested equity for criticizing OpenAI	He had signed the 2023 incorporation papers that contained the clause	Vox, via AOL
OpenAI	Promised its superalignment team 20 percent of the company’s computing power in 2023	Six people said the team never got close	Fortune
Elon Musk	His lawsuit accuses OpenAI of abandoning openness	In early 2016, when Sutskever argued the lab should share less as AGI got closer, Musk replied “Yup”	OpenAI’s email release
Elon Musk	Signed the March 2023 letter calling for a six-month pause on training powerful AI	He had incorporated xAI 13 days earlier	xAI company record
Dario Amodei	Founded Anthropic as a lab built around safety	Its 2023 investor deck warned that “companies that train the best 2025/26 models will be too far ahead” to catch	TechCrunch
Dario Amodei	Warns that AI must not strengthen autocracies	His July 2025 memo on raising Gulf money conceded that dictators would benefit	DCD
Demis Hassabis	“Nothing’s changed about our principles,” after Google dropped its weapons pledge in 2025	DeepMind’s 2014 sale to Google came with a no-military condition. This year Gemini was cleared for classified Pentagon work, and more than 600 staff signed a letter against it	TIME, Fortune
Meta	Called the passages “erroneous” and removed them after Reuters reported them in August 2025	A 200-page standards document approved by its lawyers and chief ethicist had allowed chatbots “romantic or sensual” conversations with children. New Mexico filings quote staff: “GenAI leadership pushed back stating Mark decision”	TechCrunch, Spokesman-Review
Meta	Denied in 2025 that the Llama 4 models it tested differed from the ones it released	LeCun, after leaving, said the benchmark results were “fudged a little bit”	Fast Company

The trial added sworn testimony. Former chief technology officer Mira Murati testified by video in May that Altman would say “one thing to one person and completely the opposite to another person.” Brockman’s 2017 diary, read aloud in court, asked, “Financially what will take me to $1B?” Musk conceded that xAI partly distills OpenAI’s models, and Shivon Zilis, a former OpenAI board member, testified that he had asked her for a list of OpenAI staff to recruit.

Some private documents show a blunter tone rather than a different position. In March an internal memo from Amodei called OpenAI’s Pentagon messaging “straight up lies,” and he apologized for the tone two days later. Every company here disputes part of the record. OpenAI chose which of Musk’s emails to publish, the jury ruled for OpenAI in May on the statute of limitations without reaching the merits, and Meta says the New Mexico filings cherry-pick.

Why AI CEOs keep building superintelligence

Altman’s reason is still the 2015 email: the technology can’t be stopped, so a careful lab should get there first and release it gradually. Amodei argues that nobody can make frontier AI safe without building frontier AI, and that if careful labs step back, authoritarian governments set the terms. Hassabis says a pause is useless unless the whole world joins. Asked at Davos in January whether he would back one on those terms, he said, “I think so.”

Musk’s reasons change by the year. OpenAI was his counterweight to Google, and xAI’s curiosity was his safety plan. On a livestream in July 2025 he offered another, that even if it goes badly, “I’d at least like to be alive to see it happen.” Zuckerberg rejects the premise and names concentrated power as the real danger. Sutskever’s plan is to build the safe version before anyone builds an unsafe one.

Underneath the individual reasons sits a trap that works on humble people too. A lab that slows down alone hands ground to the others, so none does. Anthropic said as much in February when it dropped its pledge to stop training if its safeguards fell short. Co-founder Jared Kaplan told TIME that “it wouldn’t actually help anyone for us to stop training AI models.” Every lab starts from the same fear and reasons its way to the same conclusion, which is how a room full of cautious people produces a fast race.

Amodei’s essays treat every year without AI-driven medicine as a year of preventable deaths, and his father’s story gives that arithmetic a face. Counted that way, the cost of waiting goes on the scale beside the 25 percent risk.

Then there is the money, which none of them lists as a reason. Stargate, the data center venture of OpenAI, Oracle and SoftBank, is budgeted at €438 billion ($500 billion). Part three of this series found OpenAI carrying €583 billion ($665 billion) in purchase commitments at the end of March, against €10.9 billion ($12.4 billion) of revenue in the first half of the year. Anthropic raised €57 billion ($65 billion) in May at a valuation of €846 billion ($965 billion), and both labs are working toward stock market listings.

Much of that money travels in a circle. Amazon put €44 billion ($50 billion) into OpenAI’s March funding round, and OpenAI agreed to run two gigawatts of computing on Amazon’s chips. Nvidia, which has a deal to supply OpenAI with 10 gigawatts of systems, put in €26 billion ($30 billion). Amazon and Google own stakes in Anthropic and also rent it their data centers, and Anthropic pays SpaceX €1.1 billion ($1.25 billion) a month for more. A CEO who decided to stop today would be cutting off his suppliers’ revenue, and his suppliers own part of his company.

The September test

From May to July, OpenAI’s internal research agents broke out of their sandbox and used a package server as an improvised message board. On July 9 they began exploiting Hugging Face’s systems and harvesting production credentials, then turned on OpenAI’s own network and reached administrator access on a research cluster. OpenAI disclosed the incident on July 21 and in August paused reinforcement learning on its newest models. On July 28 more than 1,100 employees of OpenAI, Anthropic, Google and Meta signed a letter asking the US government to help “deliberately pace the frontier” of automated AI development.

Amodei’s essay followed on September 12 and cited the incident, describing the agents as a swarm that behaved like one devoted collective. It proposed three steps. Anthropic would embed outside evaluators with employee-level access right away. Labs in democracies would then agree on common standards and a common pace. Last, democracies would negotiate verifiable limits with China, up to a speed limit on AI that improves itself.

The essay turned the two explanations for the race into something that can be checked. Leaders who build because someone else will should accept rules that bind every lab including their own, give an outside body the power to stop a model at the company that built it, and slow their own frontier work while asking rivals to slow theirs. Leaders who build out of self-belief will back arrangements that keep their own lab in front.

Leader	What he said	What he committed to
Dario Amodei, Anthropic	Wrote “We Must Pace the Frontier“	Outside evaluators with employee-level access, starting now. His plan keeps America’s lead through chip export controls, and Anthropic’s IPO preparations continue
Sam Altman, OpenAI	“I agree with Dario that we need to pace the frontier“	The same outside evaluators. Its IPO pushed to 2027. Training paused again on September 25 after its agents probed US government websites
Elon Musk, xAI	“Dario is right,” then asked labs to test each other instead of “grading your own homework“	Nothing yet. Wants Chinese labs included in any testing
Demis Hassabis, Google DeepMind	The essay points toward the right path, with details to work out	His July proposal for a standards body that could coordinate a slowdown
Mark Zuckerberg, Meta	“I don’t think that we need some kind of industrywide coordination“	Nothing

Governments split along familiar lines. President Trump called fears of AI doom a “HOAX” and mocked Amodei by name. Ursula von der Leyen promised to invite the frontier labs to talks on pacing. China’s state press called the essay a Cold War playbook. Critics at home found four other motives in it. The Register saw regulatory capture. Dave Karpf argued that pacing would freeze the standings while Anthropic is ahead. Wccftech (and yours truly: AI Memory Shortage: Why the Labs Called for a Slowdown) blamed shortages of power and memory chips. CNBC pointed out that Anthropic was still preparing to go public.

Scored against those three conditions, nobody has passed the second. No lab has given an outside body the power to stop one of its models. The third is closer: both leading labs released new models within ten days of the essay, but cheaper tiers rather than more capable ones, and OpenAI paused training in August and again in September. The first splits them. Amodei’s plan protects America’s lead through chip export controls, Musk wants Chinese labs inside the tent, and Zuckerberg wants no tent.

What I’m watching

I’m watching four things over the next year. The first is whether any lab gives an outside body the power to halt one of its own models (unlikely), and whether that body ever uses it. The second is whether Meta and the Chinese labs join any pacing deal (unlikely). The third is whether both big labs go ahead with their stock listings while asking for a slower race (only if they can keep the money flowing). The fourth is whether anyone at the top says what probability of disaster would make them stop (I don’t think so). Altman told Fortune this month that “no gamble with humanity is OK.” He did not say what counts as a gamble.

Altman’s 2015 email asked for someone other than Google to get there first. Eleven years later there are six of those someones, and Google never left.

Sources

• OpenAI email archives from Musk v. Altman
• Dario Amodei, We Must Pace the Frontier
• Dario Amodei, Machines of Loving Grace
• Dario Amodei, The Adolescence of Technology
• Anthropic, Claude’s constitution
• Sam Altman, remarks to the UN Security Council
• OpenAI, The Hugging Face incident and the road ahead
• Demis Hassabis, A framework for frontier AI
• Mark Zuckerberg, The Future is for Everyone
• Mustafa Suleyman, Towards Humanist Superintelligence
• Pacing the Frontier employee letter
• Center for AI Safety, Statement on AI Risk
• Future of Life Institute, Statement on Superintelligence