“Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.” - Sam Altman, CEO of OpenAI
“AI has the potential of civilizational destruction.” - Elon Musk, CEO of X
“I'm optimistic on the p(doom) scenarios, but... the underlying risk is actually pretty high.” - Sundar Pichai, CEO of Google/Alphabet
“I've not met anyone in AI labs who says the risk [from training a next-gen model] is less than 1% of blowing up the planet.” - Jaan Tallinn, a lead investor in Anthropic
We deeply needed full-length, thoughtful public statement from a leading frontier model company CEO acknowledging the risks of developing superintelligence. In this vein, Anthropic CEO Dario Amodei’s new essay is incredible.
However, I think there are two issues with it. One is more of a meta/intro issue, and one is a more serious critique:
(1) It maintains a self-contradictory blanket rejection of doomerism
Amodei opens by explicitly rejecting “doomerism,” which he characterizes as quasi-religious thinking. Yet his own essay:
Begins with and is named after the Contact opening: “How did you do it? How did you evolve, how did you survive this technological adolescence without destroying yourself?”
Describes AI as “the single most serious national security threat we’ve faced in a century, possibly ever.”
Reminds the reader that AI-assisted, bioengineered mirror life could “crowd out all life on the planet, in the worst case even destroying all life on earth.”
And essentially concludes with: “Whether we survive that test and go on to build the beautiful society described in Machines of Loving Grace, or succumb to slavery and destruction, will depend on our character and our determination as a species, our spirit and our soul.”
Moreover, at the 2025 Axios AI+ Summit, Amodei said:
“There’s a 25% chance that things go really, really badly.”
The distinction between how he feels about it and how doomers feel about it thus appears to be mostly aesthetic to me.
Believing there’s a 25% chance of things going “really, really badly” is obviously doom-adjacent. Why does he believe the chances are only 25%? Are we to believe there’s a rigorous model behind this? If so, where is it? I think such a model is impossible to build. Yudkowsky lays out a pretty compelling argument for nearly 100% doom in If Anyone Builds It, Everyone Dies. I don’t believe this is true, but, like everyone else who doesn’t believe this is true, some part of the logic is “I really just don’t think that will happen.” This feels like a dangerous bias to me.
Amodei seems to want the social credibility of being “measured” and “pragmatic” while making arguments that, on their merits, would justify a pretty high p(doom). I think this might be a good idea, to be clear. The public tends not to be very receptive to CEOs who believe there’s a high probability their product will kill everyone. However, Anthropic’s whole premise, from my perspective, is that they believe this and think that a ton of dedicated researchers working on a maximally aligned model is the best way to handle the threat. (This is coherent.)
(2) It places undue weight on the likelihood of our ability to avoid a biosecurity catastrophe
It is great that Amodei sees biology as the “most serious vector of attack.” He offers three categories of solutions to the biosecurity problem:
International cooperation and treaties
Guardrails on models
Investment in biodefense and preparedness
We should pursue all three. However, below is why I believe (3) must be the priority, as well as why I would categorize (1) and (2) as merely time-buying measures to solve (3).
International Cooperation and Treaties
In the section on international cooperation, he acknowledges that international cooperation and treaties are difficult. However, in the next breath, he notes there’s precedent in the Biological Weapons Convention (BWC) and that “even dictatorships do not want massive bioterrorist attacks.”
I find this to be a completely ridiculous example. In 1971, while the BWC was still being negotiated, the Soviets started planning a massive expansion of their offensive biological weapons program. They signed the BWC in 1972. In 1974 they created Biopreparat, a “civilian” biotech. This “civilian” biotech and associated bioweapons programs employed a massive 30,000 to 60,000 people across ~50 dual-use research and production facilities. They weaponized anthrax, plague, and smallpox. More importantly, they were working on intelligently designed bioweapons:
They developed antibiotic-resistant strains using genetic engineering
They built production capacity for hundreds of tons of biological material annually
There is strong evidence that they were:
Working on engineering smallpox-ebola hybrids and other chimaeras
Working on engineering bacteria to produce viruses
Here is a small excerpt on defector descriptions from one of the papers on this to emphasize just how bad these programs were:
Genes from Ebola, encephalomyelitis, and smallpox viruses were inserted into the plague bacteria. Anyone infected with these engineered plague strains would be given antibiotics to knock out the bacterial infection, but a viral infection would explode a few days later. Soviet researchers also sought to create strains to suppress the human immune system and to transform nonpathogenic microorganisms into lethal ones.
For twenty years, the Soviets denied this. We only found out about it when defectors like Vladimir Pasechnik and Ken Alibek revealed the program’s scope in the early 1990s.
After they admitted to a violation and promised to shut everything down, key military facilities in Kirov, Yekaterinburg, and Sergiev Posad remained off-limits to outside inspection. They still are.
The State Department still believes that Russia maintains an offensive bioweapons program in violation of the BWC. Moreover, they publicly acknowledge their assumptions that China, Iran, and several other countries also maintain offensive bioweapons programs.
Maybe a better-designed, less lenient treaty would make a difference, but the BWC is a categorically horrible example of what we should be aiming for. In the long-run, I am highly skeptical that any treaty would be effective at prevention.
I do think there is a chance it buys us what we really need: time to prepare. I therefore think that signing a strong one would be good for pandemic preparedness.
Guardrails on Models
Amodei is honest about the limits with guardrails, describing the obviously true “prisoner’s dilemma” framework where companies might defect from safety measures to lower costs or make their models more useful. He therefore calls for regulation to make these practices mandatory. I strongly aver that even perfect laws and perfect compliance from every Western AI company wouldn’t solve the problem, and that we cannot rely on this type of cooperation from the East. The prisoner’s dilemma framework also applies to nation states.
China is the second most capable AI developer in the world. There is no meaningful mechanism to require Chinese AI companies to implement bioweapons safeguards and I don’t believe there is any treaty or geopolitical pressure that would (to them) outweigh the strategic value of the freedom to develop AI without guardrails. It’s possible this isn’t true, and we could get an agreement in place that both parties were serious about, but we would still face a strong likelihood of one party defecting. The are also many open-source models coming out of China, which compounds the problem, as they make jailbreaking especially viable.
I am not arguing against implementing safeguards. Anthropic and OpenAI in particular have done tremendous work to this end. Safeguards are great for the same reason treaties are. Again, they buy us time.
We are incredibly lucky that the best people at making models also happen to care about this. I just think we should be realistic about how long the line can hold. (Not to mention that we don’t even need AI uplift for a sophisticated bioengineered threat to lead to catastrophe.)
Investment in Biodefense and Preparedness
We cannot rely on keeping Pandora’s box closed. I believe civilization as we know it depends on how prepared we are when it opens.
Amodei makes a critical point about this third category:
“…unfortunately I think our expectations on the defensive side should be limited. There is an asymmetry between attack and defense in biology, because agents spread rapidly on their own, while defenses require detection, vaccination, and treatment to be organized across large numbers of people very quickly.”
I think including this was a bad idea. The same logic also undermines the reliability of treaties and guardrails. In fact, it is the very logic I just used to argue that we cannot fully, or even mostly, rely on those things. The difference is that I think some type of bioengineered threat, AI-assisted/directed or not, is likely inevitable. If people believe we can prevent but cannot prepare, we will be unprepared when it comes.
Asymmetry is a major issue with preparedness, but it is one we have to try to solve. We must do everything we can.
For my thoughts on how to solve this and what some of these measures should actually look like, please see my Notes on a Defense Stack for AI-Enabled Biowarfare and Bioterrorism. (The purpose of this post is not to opine on this so-much-longer-I-could-write-a-book topic.) One small caveat is that I have learned a lot since I wrote it, and hope to update it/write new pieces with more ideas in the future.
I believe this is one of the most critical existential threats facing mankind, and that preparedness is the best way to stop it. If this is of interest to you, I highly recommend looking into BlueDot Impact’s Biosecurity course. I have worked with them on some field-level strategies in relevant areas and believe their knowledge-base and ecosystem are excellent.
Disclaimer: This piece represents my personal views and my personal views only. It does not represent the opinion of any organization I work for or with.
Didn't expect such a clear analisis of that doomerism paradox; it's realy important you've pointed out the intellectual gymnastics these leaders perform, acknowledging huge risks while dismissing the 'doom' label.
Even more good stuff, thank you.
Let's take your wise suggestion about preparation to the next level.
We should be prepared to lose. It's not going to be possible to align AI, because AI is not one thing as so many seem to assume. It increasingly will be a thousand things run by thousands of people with thousands of agendas. There is simply no way to really control AI unless we could control all the people who will be using AI, a prospect which seems extremely unlikely.
Surely we should focus on bio defense as you suggest. We should hope and pray for success. And be prepared for defeat. A single simple sentence could possibly help in that regard.
There is no proof that life is better than death.
What we're worried about is death, mass death. We typically assume that death is the ultimate tragedy, and fight against it based on that assumption. But our assumption that death is bad is really based on little more than ignorance and fear. Maybe death is defeat, maybe not, we don't really have any idea.
And there is a growing body of evidence from near death experiences that death MAY be (emphasis on MAY) be the most wonderfully spectacular experience we'll ever have. There's no way to prove any claim on the matter, but part of being prepared for AI based biological attacks could be to keep an open mind on the subject.