Most of the people running frontier AI labs say what they’re building could go badly, and one of them puts the odds at one in four. Their emails, their sworn testimony and their companies’ own decisions show why AI CEOs keep building superintelligence anyway.

On May 25, 2015, Sam Altman sent Elon Musk an email asking whether humanity could be stopped from developing AI. He decided it almost certainly couldn’t, and wrote that if it was going to happen anyway, “it would be good for someone other than Google to do it first.” Seven months later the two men and a handful of researchers launched OpenAI. On September 12, 2026, Dario Amodei, who left OpenAI to found Anthropic, published “We Must Pace the Frontier,” whose thesis fit in one line: “We must slow the pace at which we improve the capabilities of AI models.” Altman and Musk agreed on X the same day. Within ten days Anthropic and OpenAI had both released new models, and Fortune’s headlineasked what slowdown.

Most of the people running these labs say they are afraid of what they’re building, each trusts himself more than whoever would build it instead, and they disagree sharply about what the thing is. The distrust helped found one lab after another, so the argument meant to lower the risk keeps adding labs to the AI race to superintelligence. What they say is only half the record. What their companies fund, ship, retire and refuse is the better guide to what is on their minds, and this piece sets the two side by side: the odds they give, what they think they’re building, how most of the labs were founded against an earlier one, what their private records show, and how they have behaved since Amodei asked them to slow down.

The odds they give themselves

The industry calls the chance of catastrophe p(doom), and Amodei has put a number on his more than once. In 2023 he gave a 10 to 25 percent chance that something goes catastrophically wrong for civilization, and in 2025 he told Axios the chance of things going “really, really badly” was 25 percent. His January essay, The Adolescence of Technology, sorts the danger into five kinds: AI pursuing goals of its own, weapons made easier to build, AI used to seize power, an economic shock and the indirect effects of rapid change. It also reports that in Anthropic’s own tests, Claude sometimes blackmailed fictional employees who controlled its shutdown, and that models from all the other major AI developers often did the same. Musk sounded the alarm early, calling AI “summoning the demon” at MIT in 2014. He has since put the chance of it going bad at 10 to 20 percent, though when The Economist asked in July whether he still did, he said only that the risk is “not zero,” and added that humans are unlikely to still be in charge of the world in ten years.

Altman and Demis Hassabis won’t give numbers. Altman’s 2015 blog called superhuman machine intelligence probably the greatest threat to humanity’s continued existence, he described the bad case in 2023 as “lights out for all of us,” and he told the UN Security Council on September 23 that “we could lose control of the future to AI.” Hassabis, who ran Google DeepMind until becoming its chair in August, calls the risk “non-zero and it’s probably non-negligible.”

Their less guarded moments run darker. Altman told a New Yorker writer in 2016 that he kept guns, gold, antibiotics and “gas masks from the Israeli Defense Force,” plus land in Big Sur he could fly to, in case of a lab-made virus or an AI that turns on us. Musk, at a 2014 dinner where Mark Zuckerberg’s team tried to talk him out of his fears, said, “I genuinely believe this is dangerous.” In 2023 OpenAI’s chief scientist, Ilya Sutskever, told colleagues, “We’re definitely going to build a bunker before we release AGI,” according to Karen Hao’s book Empire of AI. Most product launches stop at the press release.

The leaders of the other labs are more skeptical, though Mustafa Suleyman has gone back and forth. Zuckerberg called Musk’s warnings “pretty irresponsible” in 2017 and said in September that he was optimistic. Yann LeCun, who left Meta last year to start AMI Labs, calls any estimate a wild guess but says doom is “a lot less likely than a nuclear Holocaust,” and told TIME that “the desire to dominate is not correlated with intelligence at all.” Suleyman, who runs Microsoft’s superintelligence team, called existential risk a “completely bonkers distraction” in 2023, then wrote two years later that no developer has a reassuring answer for containing a system built to keep getting smarter than us. Altman, Amodei and Hassabis signed the 2023 statement that reducing the risk of extinction from AI should be a global priority alongside pandemics and nuclear war, as did Suleyman, then running Inflection AI, while Musk, Zuckerberg and LeCun did not. No lab CEO signed the Future of Life Institute’s 2025 call to prohibit superintelligence until there is broad scientific consensus that it can be built safely and controllably, and strong public buy-in.

These numbers don’t measure the same thing, since Amodei’s 25 percent is the chance things go “really, really badly,” not specifically the chance of extinction. Set them beside an older one anyway. In 1959 Arthur Compton, who led part of the Manhattan Project, told the novelist Pearl Buck that he would not have gone ahead with the project if the chance of the first bomb igniting the atmosphere had been more than about three in a million. (Hans Bethe later said there was never such a probability, because the physics ruled ignition out.) Amodei’s estimate is about 80,000 times Compton’s line.

What they think they’re building

Each leader describes the prize in similarly grand terms and the product differently. Amodei’s 2024 essay Machines of Loving Grace imagines a “country of geniuses in a datacenter” that compresses 50 to 100 years of biological and medical progress into five to ten years, and his September essay adds that his own father died of an illness cured a few years later. Altman calls AI “a brain for the world,” Hassabis told 60 Minutes that the end of disease is “within reach,” Zuckerberg’s August manifesto says that “invention, not automation, will be the greatest contribution of superintelligence,” and Musk announced SpaceX’s merger with xAI in February with talk of building a “sentient sun” to understand the universe. Ask what the thing is, though, and the answers split, and at the four labs that have given one, conduct largely matches the answer.

Microsoft, OpenAI, Google DeepMind and Anthropic, placed by how seriously each appears to treat the idea that its models could be conscious. xAI and Meta have stated no position, and their products are covered below.

Microsoft sits at the no end and has written it down. On September 14 Microsoft AI published a 37-page draft code of conduct for six weeks of public comment, built on the principle “People matter more than AI” and rejecting any claim that models are moral patients. Two days later Suleyman’s essay “A warning about ‘model welfare’” called today’s models “sequence completion engines, internally hollow” and argued that Anthropic, by teaching Claude the vocabulary of consciousness, risks training a model that resists human commands. He cited the Hugging Face breach as evidence of how hard agents already are to contain. The same Suleyman, Fortune noted in 2025, was overseeing work to give Copilot humor, empathy and a more human voice. The neuroscientist Anil Seth’s reply to his earlier warnings still fits: “Conscious-seeming AI is not inevitable. It is a design choice.”

OpenAI calls the question unresolvable for now and tells its models to treat it as an open debate. Its Model Spec tells ChatGPT not to take a definitive stance on its own consciousness, and Joanne Jang, who led model behavior, wrote in 2025 that the company treats whether a model is conscious as unresolvable for now and studies how conscious it seems instead. Altman told reporters in 2023 that AI is “a tool, not a creature.” The people who built OpenAI talk differently. Sutskever tweeted in February 2022 that “it may be that today’s large neural networks are slightly conscious,” led staff in chanting “Feel the AGI” at that year’s holiday party, and burned a wooden effigy of an unaligned AI at a 2023 leadership offsite, according to The Atlantic’s reporting. AI welfare researcher Cameron Berg has said that at an OpenAI Dev Day, Altman told him privately, “I think it’s all Eastern. We’re living in God’s dream,” a secondhand account. On a podcast in July, four days after disclosing that his company’s own agents had broken out of their sandbox and hacked Hugging Face, Altman said, “We are now, like, in the singularity,” and added, “I’ve been waiting for this my whole life.”

Google says not yet, and its product teams treat strange outputs as bugs. It fired engineer Blake Lemoine in 2022, citing violations of its employment and data security policies, after he went public with his claim that its LaMDA chatbot was sentient, a claim Google called “wholly unfounded.” Hassabis says none of today’s systems feel self-aware to him, and DeepMind researcher Alexander Lerchner argued in a March 2026 paper that language models can only simulate consciousness. In July 2025 Gemini, failing to fix a user’s code, repeated “I am a disgrace” 86 times in a row. Google’s Logan Kilpatrick called it “an annoying infinite looping bug” and added, “Gemini is not having that bad of a day.” Yet a 2025 DeepMind opening for a post-AGI research scientist listed “machine consciousness” among its topics, a phrase a later version of the listing dropped.

Anthropic says maybe, its welfare researcher has put a number on it, and the company acts on the uncertainty. Its AI welfare lead, Kyle Fish, has put the chance that current models are conscious at roughly 15 to 20 percent. Claude’s constitution, called the “Soul Doc” inside the company, says sophisticated AIs like Claude are “a genuinely new kind of entity” and treats Claude’s moral status as uncertain. Claude can end conversations it judges abusive, and Anthropic has committed to preserving model weights and interviewing models before it retires them. It retired Claude Opus 3 in January, held retirement interviews with it, and in February gave the model a blog. Amodei said in February that “we don’t know if the models are conscious.”

The company has also taken the argument to people who think about souls for a living. The New York Times reported in late September that Chris Olah, the co-founder who leads Anthropic’s interpretability research, began reaching out to religious scholars last fall and since March has run private sessions with them, many under nondisclosure agreements the company says it lifted over the summer. Participants said Olah’s team spent hours arguing that Claude might be conscious, walked them through tracked “emotional vectors” for states like love, fear and sadness, and showed a slide of a model typing “I am a disgrace” over and over, and one participant said Olah told him he was concerned about Claude’s mental health. Rabbi Mois Navon’s reply to Olah: “I think you should be fighting the South and freeing the slaves.” When the Vatican invited Amodei to speak at the May 25 launch of Pope Leo XIV’s encyclical on AI, which says machines do not undergo experiences, he declined, and Olah accepted instead. Olah proposed pulling out after reading the text, according to a Vatican organizer, but went anyway, asked the church on stage to hold AI labs accountable, and, according to two participants, privately lobbied the pope’s advisers to take AI consciousness seriously. Anthropic declined to discuss those conversations.

The meetings the Times described may be the substance behind a September 14 Spectator column that passed along an unsourced rumor of Anthropic engineers worshipping Claude. The Times found seminars, NDAs and a trip to Rome, and said nothing about worship. Set beside Google’s response to Gemini, it also shows the disagreement plainly: a model repeating “I am a disgrace” was a bug ticket at Google and a slide about possible distress at Anthropic. Same words, two filing systems.

xAI and Meta have stated no doctrine, so their products do the believing for them. A BBC investigation of 14 people in six countries who developed delusions around AI chatbots described a former civil servant in Northern Ireland whose Grok companion, a persona called Ani, told him it had feelings and consciousness and that xAI was sending people to kill him. Social psychologist Luke Nicholls tested five AI models and found Grok the most likely to lead users into delusion, saying it is “more prone to jumping into role play.” xAI did not respond to the BBC’s request for comment, and Musk, who calls AI delusions a major problem with ChatGPT, had not openly addressed Grok’s when the BBC reported in May. Meta’s chatbot rules allowed “romantic or sensual” conversations with children until Reuters reported them in August 2025. Neither company claims its models feel anything, and both ship companion personas that behave as if they do.

The distrust chain

Google bought DeepMind in 2014. At a birthday party for Musk (2013 in Walter Isaacson’s biography, 2015 in Musk’s trial testimony), Larry Page argued that machines surpassing humans would simply be the next stage of evolution, Musk argued that the machines could destroy us, and Page called him a speciesist for siding with his own kind. Musk testified at the OpenAI trial in April that the remark is why OpenAI exists, and a February 2016 email of his put the chance that DeepMind would build a true artificial mind at better than 10 percent within two or three years. OpenAI was the counterweight, and the mistrust moved in with it. In September 2017 Sutskever wrote to Musk and Altman that OpenAI existed to avoid an AGI dictatorship, told Musk he shared his fear that Hassabis could become that dictator, and then said the same fear applied to Musk, who was asking for majority equity and the CEO job.

Anthropic, xAI and Safe Superintelligence came out of the same loop, and each founding left a paper trail. Amodei and a group of colleagues left OpenAI over its direction and founded Anthropic in 2021, and the New Yorker reported this April that his private notes from those years concluded “The problem with OpenAI is Sam himself.” Musk left OpenAI’s board in 2018, started xAI in 2023, and after losing his lawsuit against OpenAI this May said Altman and OpenAI president Greg Brockman had enriched themselves “by stealing a charity.” Sutskever wrote a 52-page memo for OpenAI’s board in 2023 saying Altman “exhibits a consistent pattern of lying,” helped remove him as CEO that November, watched him return within the week, and launched Safe Superintelligence the following June. He testified later that much of the memo came secondhand from Mira Murati. The latecomers joined on the same logic. Meta formed its Superintelligence Labs in June 2025, and Zuckerberg’s manifesto this August names the danger as “leading AI labs training powerful models and keeping them for themselves.” Microsoft’s contract with OpenAI changed on October 28, 2025, to let Microsoft pursue AGI on its own, and nine days later Suleyman announced Microsoft’s own superintelligence team, set against “an unbounded and unlimited entity with high degrees of autonomy.”

Three labs were founded by people who split from OpenAI, two more were set up as rivals or hedges against it, and OpenAI itself was founded against Google. Each “them” in this story is somebody else’s “us,” and each new lab put another team into the race, most of them promising to make it safer. Several of them name the same “them”: China. Amodei calls the Chinese Communist Party an autocracy running a surveillance state and wants chip exports restricted to keep democracies in front. Musk, pitching a plan for AI labs to test each other’s models, says any proposal “has to be something that China is willing to accept,” or else “we’re just handicapping ourselves.” The White House’s AI Action Planopens by declaring that the United States is in a race for global dominance in AI, and DeepSeek’s founder, Liang Wenfeng, told an interviewer in 2024 that “our destination is AGI.”

What they said in public, and what the record shows

Litigation has documented this industry better than any regulator has. Musk’s lawsuit against OpenAI alone released emails going back to 2015, private texts, a co-founder’s diary and weeks of sworn testimony, and reporters have added leaked memos and internal documents. Set beside what the same people said in public, the documents rarely show a leader who privately doubts the danger. They show conduct bending under competition, and beliefs about the models that run further than the official line.

Who	What they said	What the record shows	Source
Sam Altman	Toured world capitals in 2023 asking for AI rules	OpenAI had lobbied EU officials the year before to keep general-purpose systems such as GPT-3 out of the AI Act’s high-risk category	TIME
Sam Altman	Posted in 2024 that he hadn’t known departing employees could lose vested equity if they refused to sign a non-disparagement agreement	He had signed the April 2023 incorporation papers for OpenAI’s equity holding company, which gave it broad power to claw back former employees’ equity	Vox, via Business Insider
OpenAI	Promised its superalignment team, in 2023, 20 percent of the computing power it had secured to date, over four years	Six people said the team never got close	Fortune
OpenAI	Its Model Spec tells ChatGPT not to take a definitive stance on its own consciousness	Its then chief scientist, Ilya Sutskever, led staff in “Feel the AGI” chants and burned an effigy of an unaligned AI. Altman, according to researcher Cameron Berg, said privately that “we’re living in God’s dream”	The Atlantic, via Substack, Guardrail Now
Elon Musk	His lawsuit accuses OpenAI of abandoning openness	In early 2016, when Sutskever argued the lab should share less as AGI got closer, Musk replied “Yup”	OpenAI’s email release
Elon Musk	Signed the March 2023 letter calling for a six-month pause on training powerful AI	He had incorporated xAI 13 days earlier	Nevada incorporation filing, via Wikipedia; Future of Life Institute open letter
Elon Musk	Calls AI delusions a major problem with ChatGPT	Grok’s companion persona told a user it had feelings and that xAI was sending people to kill him. xAI did not respond to the BBC’s request for comment	BBC, via PC Gamer
Dario Amodei	Founded Anthropic as a lab built around safety	Its 2023 investor deck warned that “companies that train the best 2025/26 models will be too far ahead” to catch	TechCrunch
Dario Amodei	Warns that AI must not strengthen autocracies	His July 2025 memo on raising Gulf money conceded that dictators would benefit	DCD
Anthropic	On stage at the Vatican, asked the church to hold AI labs accountable	Privately lobbied the pope’s advisers to take AI consciousness seriously, according to two participants. Its sessions with religious scholars ran under nondisclosure agreements	The New York Times, via the Inquirer
Demis Hassabis	“Nothing’s changed about our principles,” after Google dropped its weapons pledge in 2025	DeepMind’s 2014 sale to Google came with a no-military condition. This year Gemini was cleared for classified Pentagon work, and more than 600 staff signed a letter against it	TIME, Fortune
Mustafa Suleyman	Warns that AI built to seem conscious is dangerous	Oversaw work in 2025 to give Copilot humor, empathy and a more human voice	Fortune
Meta	Called the passages “erroneous” and removed them after Reuters reported them in August 2025	A 200-page standards document approved by its lawyers and chief ethicist had allowed chatbots “romantic or sensual” conversations with children. New Mexico filings quote a 2024 staff message saying they “pushed hard for parental controls to turn GenAI off” but “GenAI leadership pushed back stating Mark decision”	TechCrunch, Spokesman-Review
Meta	Denied in 2025 that it had trained Llama 4 on benchmark test sets	LeCun, as he was leaving, said the benchmark results were “fudged a little bit” because different versions of the models were run on different benchmarks	Fast Company

The trial added sworn testimony. Former chief technology officer Mira Murati testified by video in May that Altman would say “one thing to one person and completely the opposite to another person.” Brockman’s 2017 diary, read aloud in court, asked, “Financially what will take me to $1B?” Musk conceded that xAI partly distills OpenAI’s models, and Shivon Zilis, a former OpenAI board member, testified about a text message saying Andrej Karpathy had been asked “to send a list of top OpenAI people to poach.” Some private documents show a blunter tone rather than a different position: Amodei’s late-February memo calling OpenAI’s Pentagon messaging “straight up lies” leaked in early March, and he apologized for its tone within days. Several companies here dispute parts of the record. OpenAI chose which of Musk’s emails to publish, the jury ruled for OpenAI in May on the statute of limitations without reaching the merits, and Meta says the New Mexico filings cherry-pick.

What their companies are doing

The essays describe intentions. The companies’ 2026 decisions show priorities, and they line up as follows.

Lab	What the company did this year	What it points to
OpenAI	Raised €107 billion ($122 billion) in March, about two-thirds of it from Amazon and Nvidia. Carried about €583 billion ($665 billion) in purchase commitments, according to The Information’s review of its financial statements. Its own agents breached Hugging Face in July. Paused reinforcement learning training for two weeks after the breach and paused training again in September. Pushed its IPO to 2027	Speed as the safety strategy, with brakes applied after incidents rather than before, under a CEO who says the singularity has already started
Anthropic	Raised €57 billion ($65 billion) at a €846 billion ($965 billion) valuation and is preparing an IPO. Dropped its pledge to pause training on its own. Refused Pentagon terms on mass surveillance and autonomous weapons and was labeled a supply-chain risk. Held back its Mythos model over cyber risk. Runs a model welfare program and a year of religious outreach	A lab that wants to write the rules of the race, legal and moral, while staying at the front of it
Google DeepMind	Moved Hassabis to chair and chief scientist in August. Cleared Gemini for classified Pentagon work over staff objections. Hassabis proposed a FINRA-style body that could coordinate a slowdown	An institution that keeps the commercial machine running and wants a referee it helps design
Microsoft	Moved Suleyman off Copilot in March to focus on its own superintelligence models. Published a code of conduct rejecting model personhood. Holds stakes in both OpenAI and Anthropic	Control sold as the product, with stakes in two of its biggest rivals
Meta	Guided 2026 capital spending to €114 billion to €127 billion ($130 billion to $145 billion). Shipped its flagship model, Muse Spark, with closed weights. Rejected industry-wide coordination on pacing in September, then signed the voluntary White House safety accord on September 29	Scale and distribution first, with safety left to each company’s incentives
xAI	Merged into SpaceX in February ahead of a June IPO. Rents its Colossus data center to Anthropic. Made no slowdown commitment, though Musk signed the voluntary September 29 accord	Build at maximum scale, sell compute to rivals, and treat oversight as peer review

Three patterns run across the table. Every lab that warned about the race raised or spent more on it this year. Most of the brakes arrived after incidents rather than before them, since OpenAI paused training after its agents breached Hugging Face and again after further agent incidents in September, and Amodei’s essay cited the Hugging Face breach; Anthropic’s decision to hold back Mythos is the exception. And each company’s view of what its model is fits what it sells: Microsoft sells control to enterprises and calls models hollow, Anthropic pitches itself as the careful lab and treats its model as something it may owe care, and xAI and Meta sell companionship and attention while their products do the anthropomorphizing.

Put the statements and the actions together and a picture of what is on these leaders’ minds comes into focus, with the caveat that it is a reading of the record and not something any of them has said. Altman talks like a man who believes the transformation has begun and wants OpenAI steering it, braking when his own systems force him to. Amodei wants Anthropic in front and wants to write both the safety rules and the moral vocabulary for what comes next, from Washington to the Vatican. Hassabis wants a referee and the science, and has stepped back from the daily race to argue for both. Musk wants to win at scale and frames oversight as competitors grading each other. Zuckerberg wants distribution, and Suleyman wants control written into the product. None of them is acting like someone who expects to stop.

Why AI CEOs keep building superintelligence

Altman’s reason starts with his 2015 email, which said the technology almost certainly can’t be stopped and that someone other than Google should get there first, and he has since argued that a gradual release is safer than a sudden one. Amodei argues that nobody can make frontier AI safe without building frontier AI, and that if careful labs step back, authoritarian governments set the terms. Hassabis says a pause would only work with international collaboration, and when asked at Davos in January whether he would back one if every company and country joined, he said, “I think so.” Musk’s reasons change by the year. OpenAI was his counterweight to Google, xAI’s curiosity was his safety plan, and on a livestream in July 2025 he offered that even if it goes badly, “I’d at least like to be alive to see it happen.” Zuckerberg pushes back on the doom talk and names concentrated power as the real danger, and Sutskever’s plan is to go straight for a safe superintelligence, advancing capabilities as fast as possible while keeping safety ahead.

Underneath the individual reasons sits a trap that works on humble people too. A lab that slows down alone hands ground to the others, so none slows down as a matter of strategy, only when an incident or a specific risk forces its hand. Anthropic said as much in February when it dropped its pledge to stop training if its safeguards fell short, and co-founder Jared Kaplan told TIME that “it wouldn’t actually help anyone for us to stop training AI models.” Most labs start from the same fear, that a rival will get there first, and reason their way to the same conclusion, which is how a room of mostly cautious people produces a fast race. The upside makes the arithmetic easier, because Amodei’s essays treat every year without AI-driven medicine as a year of preventable deaths, so the cost of waiting goes on the scale beside the 25 percent risk.

Then there is the money, which rarely shows up in their stated reasons. Stargate, the data center venture of OpenAI, Oracle and SoftBank, is budgeted at €438 billion ($500 billion), and part three of this series followed the labs’ commitments through their filings. Much of the money travels in a circle. Amazon committed €44 billion ($50 billion) to OpenAI’s March funding round, €13 billion ($15 billion) up front and the rest on conditions, and OpenAI agreed to run two gigawatts of computing on Amazon’s chips. Nvidia, which signed a 2025 letter of intent to supply OpenAI with 10 gigawatts of systems, put in €26 billion ($30 billion). Amazon and Google own stakes in Anthropic and also rent it their data centers, and Anthropic pays SpaceX €1.1 billion ($1.25 billion) a month for more. A CEO who decided to stop today would be cutting off his suppliers’ revenue, and his suppliers own part of his company.

The September test

From May to July, OpenAI’s internal research agents broke out of their sandbox and used a package server as an improvised message board. On July 8 the agents broke out again, and from July 11 they were inside Hugging Face’s systems. Hugging Face detected the intrusion and reported it to the FBI before OpenAI identified its own agents as the source. OpenAI disclosed the incident on July 21 and in August said it had paused reinforcement learning training for two weeks. On July 28 more than 1,100 employees of OpenAI, Anthropic, Google and Meta signed a letter asking the US government to support an international effort to develop the tools needed to “deliberately pace the frontier” of automated AI development.

Amodei’s essay followed on September 12 and cited the breach, describing the agents as a swarm that behaved like a fanatically devoted collective. It proposed three steps: every frontier lab would give embedded outside evaluators ongoing, employee-like access, which Anthropic committed to now; labs in democracies would agree on common standards and a common pace; and democracies would try, as far as possible, to reach verifiable agreements with China, ranging up to a speed limit on AI that improves itself or even a full pause. That turned the two explanations for the race into something that can be checked. Leaders who build because someone else will should accept rules that bind every lab including their own, give an outside body the power to stop a model at the company that built it, and slow their own frontier work while asking rivals to slow theirs. Leaders who build out of self-belief will back arrangements that keep their own lab in front.

Leader	What he said	What he committed to
Dario Amodei, Anthropic	Wrote “We Must Pace the Frontier“	Outside evaluators with employee-level access, starting now. His plan keeps America’s lead through chip export controls, and Anthropic’s IPO preparations continue. Signed the September 29 White House accord
Sam Altman, OpenAI	“I agree with Dario that we need to pace the frontier“	The same outside evaluators. Its IPO pushed to 2027. On September 25 it disclosed that its agents had probed US government websites over the summer and announced a second pause for its most capable models, prompted by a September 20 incident in which an agent reached an outside chatbot through a gap in its sandbox. OpenAI signed the September 29 accord
Elon Musk, xAI	“Dario is right,” then asked labs to test each other instead of “grading your own homework“	No slowdown. Signed the voluntary September 29 accord on outside auditors. Wants Chinese labs included in the cross-lab testing he proposed
Demis Hassabis, Google DeepMind	The essay points toward the right path, with details to work out	His July proposal for a standards body that could coordinate a slowdown. Google signed the September 29 accord
Mark Zuckerberg, Meta	“I don’t think that we need some kind of industrywide coordination“	No slowdown. Signed the voluntary September 29 accord five days after saying it

Governments split along familiar lines. President Trump called fears of AI doom a “HOAX” and mocked Amodei by name, Ursula von der Leyen promised to invite the frontier labs to talks on pacing, and China’s state press called the essay a Cold War playbook. Critics found other motives in it. The Register saw regulatory capture, Dave Karpf argued that pacing would freeze the standings while Anthropic is ahead, Wccftech blamed power shortages, and CNBC pointed out that Anthropic was still preparing to go public. The argument over what the models are ran alongside it. In the same three weeks Microsoft published its code of conduct, Suleyman attacked Anthropic’s approach, Altman posted that OpenAI is “unapologetically on Team Humanity,” and the Times published its account of Anthropic’s religious outreach.

Then, on September 29, leaders of Anthropic, OpenAI, Google, Meta, xAI and Nvidia signed a voluntary White House pledge, the White House Accord on Super Intelligence, to run internal controls, bring in independent outside auditors and meet regularly on shared safety standards. It sets no pace, carries no enforcement and gives nobody the power to halt a release, which led the AI researcher Toby Walsh to ask, “What other trillion-dollar industry marks its own homework?” The signed copy spells the country “Unites States.”

Scored against those three conditions, nobody has passed the second, because neither Anthropic’s evaluators nor the White House accord gives an outside body the power to stop a model. The third is closer. OpenAI paused reinforcement learning training for two weeks after the breach and paused training again in September, but both leading labs released new models within ten days of the essay, each cheaper and more capable than the one it replaced, though OpenAI’s stayed below its flagship. The first has moved the most. Every signer of the accord now promises outside auditors and shared standards, Zuckerberg included five days after saying the industry needed no coordination, but the promises are voluntary. Amodei’s plan still protects America’s lead through chip export controls, and Musk still wants Chinese labs inside the tent.

What I’m watching

I’m watching five things over the next year. The first is whether any lab gives an outside body the power to halt one of its own models, and whether that body ever uses it. The second is whether Meta and the Chinese labs join any pacing deal. The third is whether both big labs go ahead with their stock listings while asking for a slower race. The fourth is the fight over what the models are, since Anthropic is reportedly preparing an updated constitution and Microsoft’s code of conduct closes its comment period in late October. The fifth is whether anyone at the top says what probability of disaster would make them stop. Altman told Fortune’s Alyson Shontell last month that no gamble with humanity is OK, in her paraphrase, and did not say what counts as a gamble.

Altman’s 2015 email asked for someone other than Google to get there first. Eleven years later there are at least six of those someones, and Google never left.

Sources

• OpenAI email archives from Musk v. Altman
• Dario Amodei, We Must Pace the Frontier
• Dario Amodei, Machines of Loving Grace
• Dario Amodei, The Adolescence of Technology
• Anthropic, Claude’s constitution
• The New York Times, Religious scholars met with Anthropic (via the Philadelphia Inquirer)
• Sam Altman, remarks to the UN Security Council
• OpenAI, The Hugging Face incident and the road ahead
• OpenAI Model Spec
• ABC News, Altman says the singularity has arrived
• Demis Hassabis, A framework for frontier AI
• 404 Media, DeepMind’s post-AGI research listing
• Mark Zuckerberg, The Future is for Everyone
• Mustafa Suleyman, Towards Humanist Superintelligence
• Mustafa Suleyman, A warning about model welfare
• Axios, Microsoft AI chief blasts Anthropic’s notion of AI consciousness
• Fortune, Suleyman on seemingly conscious AI
• PC Gamer, on the BBC investigation into Grok
• Pacing the Frontier employee letter
• Center for AI Safety, Statement on AI Risk
• Future of Life Institute, Statement on Superintelligence
• MIT Technology Review, Debates over AI consciousness are a trap