A few weeks ago, we had an argument online that garnered quite a bit of attention. We spoke for a long time afterwards, as many people who disagree should, and discovered the crux of the issue: we disagree significantly on what artificial intelligence progress looks like.
More importantly, we discovered there is a lot we agree on.
We agreed that biological risk is real, old, tractable, and not mainly an AI-policy problem. We agreed that we should not react with fear, and that it is the job of policymakers and technologists to make bioweapons a poor choice for adversaries.
We decided to write this piece together to explain why, and some recommendations we agree on.
Biosecurity has Mattered for Decades
The central reason why differing opinions on progress in artificial intelligence do not lead to differing opinions on what we should do about biological threats today is that we both agree bioweapons are already a threat now, and have been a threat for decades.
US intelligence is public about ongoing state programs dedicated to the development of bioweapons. The history of state bioweapons programs, particularly the history of the Soviet program Biopreparat, which employed 30,000 people and stockpiled smallpox by the ton, should dispel notions that state actors would not pursue this type of war.
There have also already been multiple attacks by non-state actors to date, both successful and unsuccessful. Though disputed, it is even possible that we have already seen a successful “lone wolf” bioattack in the form of the anthrax letters that killed 5 Americans and sickened 17 others after 9/11.
The role of Artificial Intelligence and the Culture War
Even where there was more disagreement, we still found common ground. Much of the debate online centered on scenarios like “superintelligent” AI” taking over a fully autonomous lab, neither of which exist yet, and designing a weapon that could kill everyone.
We posit that haggling over whether a bioweapons attack could kill everyone, half of all people, or a tenth of all people, is not a useful conversation for biosecurity. We have ample evidence from natural pandemics that it is possible to wipe out a significant proportion of a population. We do not need to agree on the right tail to build defenses that work across the distribution, which should have been built decades ago.
Regarding biological models, we agree that a model can propose a genome that is synthesizable and, if it is close enough to something known, infectious. We also agree that this is not the same thing as designing a human-extinction level pandemic, and that wet lab work still takes time, equipment, and iteration in both cell and animal models.
We agree that biological safeguards on AI models are not sufficient to reduce biological risk, primarily because open weight models are less than a year behind and can be run privately. We agreed that hosted inference, such as on OpenRouter, can and should be monitored for bioweapons misuse.
It also turned out that we mostly agree on what we should do about AI safety. Primarily we agree that it is important to do safety engineering on models by deciding how we want these systems to behave, imposing this on the system, then testing to ensure this safety engineering is effective.
The culture war around AI appears to only be growing more and more extreme. It is important not to let this stop us from building practical things that we can all agree on, including in the realm of biosecurity.
Various biodefense measures have had Republican and Democratic sponsors for years. The requirement for nucleic acid (DNA & RNA) providers to monitor the sequences that customers order, for instance, was first put forward during the Bush admin. It was again put forward during the Biden admin. It is now a bicameral bill under consideration during the Trump administration. Republicans and Democrats align almost completely on biosecurity. (In fact, some of these efforts are so noncontroversial that it is difficult to prioritize them, because no one sees them as a win for their side.)
Incentives in Biowarfare
Why would anyone develop a bioweapon anyways? They have significant disadvantages. It is difficult to make them targeted. From a state perspective, one must also protect their own population. This is why the Soviets stockpiled vaccines against their own weapons during the height of Biopreparat. However, there are three reasons an adversary might make bioweapons their weapon of choice: their potential for destruction, ambiguous attribution, and the general offensive advantage.
Bioweapons are self-replicating. SARS-CoV-2, a relatively tame virus, killed ~27 million people. Other weapons of this level of destruction require significantly more capital investment and personnel. Aircraft carriers cost thirteen billion dollars, for instance, and nuclear weapons programs cost tens of billions and require industrial-scale enrichment infrastructure that is visible from orbit.
In contrast, infrastructure for bioweapons development is roughly the same as infrastructure required for basic biological research. It can be successfully hidden, making attribution difficult. This is sometimes desirable. The debate the world saw around SARS-CoV-2 origins is an excellent example of how difficult it is to tell from sequence whether or not something might be engineered, let alone whether it might have come from laboratory samples or not.
Biowarfare has an offensive advantage, particularly with respect to time and cost. A pathogen engineered in a lab can be released immediately in target areas where it is likely to spread quickly. It does not need to be lab-tested for safety or efficacy. If it is not efficacious, the adversary can release a new one. Even if it did take a year to design a new pathogen, the clock for medical countermeasures starts after the pathogen is detected as a threat, and medical countermeasures have to go through expensive clinical trials. Lab biology is comparably cheap, and getting cheaper daily. Benchtop DNA synthesizers now cost less than many cars. In 2017, Canadian researchers successfully synthesized horsepox, a large and complex virus similar to smallpox, using mail-order DNA for approximately $100,000 USD. The lab techniques are also not as inaccessible as one might think. A graduate student can learn to synthesize a virus in roughly a couple of weeks. Biological defense, on the other hand, remains expensive. Biosurveillance networks, stockpiles of medical countermeasures and respirators, basic research, and warm manufacturing capacity for large-scale countermeasure deployment cost billions of dollars annually. Moreover, they must be maintained continuously against a threat that has not yet been deployed.
There is one area where defense has an advantage in biowarfare that rights some of this asymmetry: resources. Defenders generally have more access to capital, research infrastructure, and capable personnel. The US’ private and public sectors should use this advantage to prevent, detect, withstand, and combat engineered bioweapons, ultimately deterring their development and deployment. This is how we make biowarfare defense-dominant.
Defense in Depth
No single intervention is going to eliminate bioweapons risk, but defense in depth can. We would like to put forward four recommendations for this defense stack out of many that we agree on.
The first recommendation is clean air infrastructure. We started cleaning our water and waterborne disease (e.g. cholera, typhoid) basically disappeared from the West. We can do the same thing for air. This infrastructure would combat all respiratory disease, making it significantly more difficult to build an effective bioweapon and massively deterring potential attackers. Moreover, over 100,000 people die of respiratory viruses in the US every year. Many of these deaths are preventable. We have the ability to deploy significantly better air filtration than we have now, and we can change policy via options like revisions to ASHRAE Standard 241 to incentivize implementation. Clean air technologies like far-UVC may also significantly decrease risk of infection through surfaces, and should be investigated thoroughly. Clean air indoors is largely a public good, but there are also market reasons to do this. Clean air could reduce school absences, save large employers millions of dollars annually, and improve livestock safety.
The second recommendation is synthesis screening. There are currently more regulations on sandwiches than nucleic acid orders. Mandating synthesis screening would require providers to check who is ordering the sequences (like KYC systems in finance) and whether the sequences are dangerous, like smallpox. KYC can be massively sped by artificial intelligence, and screening for known dangerous sequences is cheap. Such a regulation should also check for split-orders, where one person might order a partial genome from one provider and the rest from another, as well as mandate that benchtop synthesizer manufacturers monitor and record their use. This is such common sense that most providers already do this of their own volition, and support the requirement. This is also exactly the kind of regulation that can prevent us from needing stricter regulations later. There is a risk of an attack leading to regulatory backlash that would slow vital biotechnology innovation.
The third recommendation is to improve and deploy metagenomic detection systems. The CDC should fund a national pathogen early-warning network to detect emerging outbreaks early. Allocating ~$80M annually would integrate metagenomic sequencing into wastewater and clinical surveillance systems to identify novel pathogens rapidly. Concurrently, the Defense Innovation Unit should continue to sponsor advanced AI tools to analyze these massive datasets and advance biosurveillance capabilities as well as deploy detection systems. Private sector organizations and companies could also take this development on.
The final recommendation is to sustain stockpiles. If you are prepared for the threat, this significantly deters the likelihood of an attack. We have let our reserves dwindle before. During the 2009 swine flu pandemic, SNS gave out tens of millions of N95 masks and never replaced them. We should provide sustained funding for the Strategic National Stockpile (SNS) to maintain reserve supplies of respirators and treatments, alongside warm-base manufacturing facilities for rapid countermeasure production. Policy researchers should consider what types of market shaping can be done to better incentivize commercial stockpiling, like agreements with the government not to seize stockpiles during an emergency except at a premium.
The amount of capital these interventions cost is trivial compared to the cost of a pandemic or what we spend on threats like nuclear. Pandemic-scale biological events could easily impose US losses in the $5-20T range. SARS-CoV-2 US losses alone exceeded $15T, with some estimates closer to $20T. Assuming a conservative 1-2% annualized probability of a natural pandemic of COVID-19 magnitude or larger, expected US losses per year are roughly $150-400B. In the lower end of that range, with loss estimates of ~$200B per year, a $5-10B/year mitigation investment is only about 2-5% of expected annual loss.
Concluding Thoughts and About Us
We should not be dismissive of or fear-monger about biological risk, nor should we be scared to bring automation to the lab. Biodefense should be maximally concentrated on things like blocking physical chokepoints, systems to detect threats immediately, and building out infrastructure that prepares us to withstand an attack so well that it deters any actor from trying.
We come from different, overlapping backgrounds that make us uniquely qualified to think about this issue.
David Bellamy is one of the few people who has both trained a frontier LLM and synthesized a virus de novo in the lab. He currently works on agentic reinforcement learning training infrastructure at MBZUAI’s Institute of Foundation Models. He was the founding Research Scientist at Lila Sciences, a multi-billion dollar Flagship Pioneering startup focused on using AI to conduct scientific research autonomously. He holds a PhD in Biostatistics and Epidemiology from the Harvard School of Public Health, where he worked on Causal Inference and Deep Learning for Medicine.
Olivia Scharfman is an expert in biosecurity and biotechnology innovation policy at the Institute for Progress (IFP), a non-partisan think tank dedicated to scientific, technological, and industrial progress. Prior to joining IFP, she co-founded LincSwitch Therapeutics, a long non-coding RNA epigenetic editing company. Before that, she was a Principal at The SALT Fund, investing in biotechnology and deep tech startups such as General Biological and Northwood Space. Her earlier work included time with convolutional neural networks and fluid dynamics at DEKA Research & Development, as well as research for XPRIZE. She holds a B.S. in Molecular Biophysics and Biochemistry from Yale.
We still disagree about the extent by which artificial intelligence uplifts biological risk, and you can too. We argue that this does not change the urgent need for building better biodefense infrastructure.
We must not let a culture war stop us from doing practical things for public safety and security.
Disclaimer: This piece represents the personal views of Scharfman and Bellamy only. It does not represent the opinion of any organizations either of us work for or with.