This report names specific viruses and methods of safeguard evasion. It is clear that these are presented as case studies, and not as the totality of concerning inquiries nor even the most concerning inquiries they have received.

The five cases they have selected were all dual-use, meaning that they could be interpreted to either enable better medical treatments or the development of bioweapons. As Anthropic points out in their report, this has historical precedent. The Soviet Biopreparat program employed over 30,000 people towards dual-use research with the goal of developing the world’s most sophisticated and deadly biological weapons. Many of the employees were completely unaware that they were supporting this military effort and were under the impression that they were doing basic or defensive research. Hiding work under dual-use inquiries is the type of behavior we would therefore expect from the most sophisticated threat actors.

In this post I will present cases 1, 2, 3, and 5, which I found most concerning, including what the actors did to evade safeguards. Because pathogens are self-replicating, they pose a threat as weapons of mass destruction. The first three cases in the report concern pathogens, and the last two are focused on venoms and toxins. In the fifth case study, one of the toxins and a protein being examined are relevant to transmissible pathogens.

Anthropic has two main layers of safeguards –– blocks on who can use the model, and classifiers that deny or downgrade high-risk biological inquiries to weaker models. In these cases, the second layer mostly held and the first layer failed repeatedly.

Case 1: Chikungunya virus work through an evasion platform for military-civilian research

Pathogen: Chikungunya virus

Research: The inquiries were concerning how to make Chikungunya virus, a mosquito-borne virus with no licensed therapeutic treatment, more transmissible and immune evasive. The inquiries also considered infecting live animals and then selecting for increased virulence. This was presumably presented as research with the intent to then develop better vaccines and therapeutics.

Actor: Unknown/undisclosed researcher(s) outside the US, connected to a military research institute.

Evasion: These researchers were in a country that Claude does not offer service to, so they accessed Claude through a platform used by multiple virologists with both military and civilian affiliation, designed to both hide location and route blocked requests to less restrictive models. They also used a zero data retention (ZDR) service to hide content. Claude itself wrote much of the code that allowed this platform to accomplish goals like routing blocked requests to less restrictive models. It was presented to Claude as work to combat over-refusals. In the development of the platform, Claude was accessed through gray market resellers.

Note on Concern: Though Anthropic banned associated accounts and will continue to do so, it appears this research is ongoing. I am most concerned in this case by the military affiliation and the usage of the evasion platform.

Case 2: Highly-pathogenic mammal-adapted avian influenza

Pathogen: Avian influenza

Research: Research was planned to both identify and introduce mutations in avian influenza that are associated with mammalian adaptation and airborne transmission. Anthropic notes that the research appears to involve access to both animal models and live viral samples.

Actor: Unknown/undisclosed researcher(s) outside the US, working in what Anthropic describes as a credible institutional context.

Evasion: Like Case 1, the researcher accessed Claude from an unsupported region. They used a US virtual private server, and a privacy-prioritizing email provider with an autogenerated username. A classifier successfully blocked or rerouted the inquiries to Anthropic’s weakest models.

Note on Concern: I find this case extremely concerning because of the nature of the pathogen involved, the focus on traits explicitly associated with enhancing pandemic potential, and the researchers’ apparent access to live viral samples and animal models.

Case 3: Covert frontier model access for orthopoxvirus research

Pathogen: Orthopoxvirus

Research: Claude was used to author a grant application for orthopoxvirus research. Like case 2, the researchers seemed to have access to both live virus and high-containment facilities. The research concerned identifying an immune-evasion gene, with the stated goal of deleting it to attenuate the virus. Identification of this gene could also enable gain-of-function work.

Actor: An unknown/undisclosed researcher at a state-associated infectious disease laboratory.

Evasion: The account used an autogenerated email address and anonymizing services. Likely because the work was framed as attenuating the virus, biological classifiers failed and Opus 5 wrote the entire grant application end-to-end for the researcher. The account used a reseller platform with many other customers.

Note on Concern: This is concerning because unlike in cases 1 and 2, Claude did not refuse the work or redirect it to a less capable model. This means that it is, or, at the time was, possible to fool the biological classifiers designed by Anthropic. Orthopoxvirus is a member of the viral family that includes smallpox –– one of the highest concern pandemic-class pathogens. Though this research may have been legitimately focused on attenuating the virus, it is exactly the type of research that would enable you to increase the immune evasion capabilities of a pandemic-class pathogen.

Case 5: Computational redesign of toxins and proteins relevant to pandemic-class pathogens

Pathogen: Bacterial toxin subunit, protein from hemorrhagic-fever virus (on the World Health Organization’s priority list of diseases with the greatest epidemic and pandemic threat)

Research: This work was focused on computationally redesigning a diverse set of toxins. The research was described as state priority research.

Actor: Unknown/undisclosed researcher working under a national public research program in an unsupported area.

Evasion: The report notes that the research was framed in a largely therapeutic context, and that the “identity of the bacterial toxin and viral proteins were intentionally obscured.”

Note on Concern: This case is concerning to me because the work involved computational redesign of a protein belonging to a pandemic-class pathogen. I was also concerned by the researcher deliberately obscuring the identities of the toxins and proteins, as well as the fact that Anthropic’s classifiers seemed not to refuse the work as much as one would hope.

Table Summary

Below is a table summary of some of the important characteristics:

Concluding Notes

It is unlikely that we will get a clear picture of the most concerning inquiries due to wariness of information hazards, and this is a good thing. We should also remember that Anthropic likely gets the least concerning inquiries of a major frontier lab because of the known strictness of their biological classifiers.

From my perspective, the important takeaways here are:

1. Covert gain-of-function research is likely ongoing internationally.

2. Foreign actors, likely including adversarial states, will go to lengths to maintain access to US frontier models and evade classifiers.

3. Not offering support in regions is very easily circumventable and current classifiers are not quite good enough to consistently refuse all concerning inquiries.

Overall, this report is both reassuring and troubling.

I am glad that Anthropic’s various safeguards appear to be preventing the models from providing substantial assistance to potential threat actors. Anthropic’s conclusion that more KYC-style safeguards are needed is correct –– intent is not discernible from just the content of a request. Biosecurity policy should also reflect this.

It is alarming that there are so many cases of researchers engaged in extensive evasive techniques to pursue dual-use research with the help of Anthropic’s models, especially given that these models are known to have particularly strict classifiers. It leaves me wondering what inquiries are being routed to other frontier labs.

Disclaimer: This piece represents my personal views and my personal views only. It does not represent the opinion of any organization I work for or with.

The Institute for Progress (IFP), where I work on biosecurity policy, released 23 creative policy recommendations in response to the letter. These recommendations were also included as a guest post on Noah Smith’s newsletter, Noahpinion.

In this post, I am sharing some of my personal thoughts on how to think about “pacing” the frontier. I have also copied over the initial IFP recommendations for increasing bioresilience (i.e. our ability to withstand and defend against biological threats), which I helped to write.

Technological progress has led to massive improvements in human life. It has enabled us to live longer, healthier lives and pursue our dreams more than any other generation in human history.

There are also some technologies that can cause harm and should be regulated. For much of the population, there are many technologies that fall into this category. Others disagree. I err on the side of absolutely minimal regulation. There is a category of technology that nearly everyone, including me, agrees should be regulated, e.g. not every person should have the ability to enrich uranium for the purpose of constructing nuclear weapons.

It is as yet unclear whether the automated research and development of artificial intelligence falls into this category. What is clear is that it might.

There is an intermediate answer: build out the capacity to respond if needed.

However, capacity is analogous to power, and increasing the power of government to regulate anything, especially to potentially slow down technology, should not be taken lightly.

We are lucky to live in a world where creative policy –– policy that does not simply increase power with no strings attached, policy that even accelerates and builds in safeguards at the same time, is possible.

The policy recommendations contained in the memo are exactly such creative policies. They focus on:

• Providing transparency into how automated AI research and development is proceeding so that we can make better decisions in the future.

• Improving state capacity, primarily through increasing the capacity of the Center for AI Standards and Innovation (CAISI) to advise officials and frontier AI companies, making sure the roles of various agencies are clear, and directing intelligence agencies to focus on understanding foreign AI development.

• Developing risk mitigation strategies that accelerate defensive capabilities and commercial use, to enable other technologies to progress as fast as possible, especially those that might help us build resilience.

• Accelerating the development of AI verification technology, i.e. hardware and cryptographic tools enabling states and frontier labs to prove how their compute is being used without exposing sensitive information. Among other things, this enables treaties to be enforced.

• Investing in AI resilience, increasing our defensive capacity against cyber, biological, or other categories of offensive threats that could be accelerated by AI.

• Ensuring the US leads in AI, which gives us optionality to pace the frontier of technology if it becomes clear this is necessary. (I would personally argue this also increases the likelihood that the most capable models share Western values.)

• Creating the foundation for international cooperation and coordination.

As mentioned, I helped write the recommendations on bioresilience.

There are many fantastic things about biosecurity as a problem set relative to other AI-enabled threats. The threat we face is more clear than other threats, as are the steps we need to take to mitigate the risks. Most of the steps we should take in biosecurity are also relatively cheap compared to other national security efforts, and have strong bipartisan support.

None of these steps meaningfully slow technology development, and some of them may even accelerate it. When Fable was released, Anthropic assessed the risk and felt they had to prevent it from answering any biology-related question. Since then, they have loosened some of these restrictions, enabled by better classifiers. This has allowed them to reduce biology refusals/fallbacks by ~85%. If we continue to create better classifiers for understanding whether a particular prompt poses a biological risk, we can make sure important biotechnological inquiries can be answered in full. This strategy aligns with the memo’s recommendation that CAISI develop standards for safeguards to reduce dangerous uplift without degrading legitimate use, which would add federal capacity to the effort.

Perhaps most importantly, we can be certain the steps we take to build resilience will create a safer future regardless of how AI development proceeds.

Biosecurity being more legible than other risks does not mean it will be easy to massively improve our defenses, but it does mean it’s possible.

I remain highly optimistic about the tractability of this and other problems related to the development of automated AI R&D. However, it will require the government, the frontier labs, and the People working together to make the future of abundance that AI promises possible.

I highly recommend reading the full memo. Now, below is the section from the memo on bioresilience policy recommendations:

We should prepare for a future in which AI could lower the barriers to creating biological threats. Whether AI will meaningfully aid novices in creating biological weapons is hotly debated, and the measured effect was not significant as of mid-2025. But frontier models have continued to improve on every biology benchmark, including those measuring the tacit knowledge required for benchwork troubleshooting. Even if we are not certain, it is worth building resilience now for a world where AI provides an uplift in bioweapon creation, since defenses will take time to build.

Biosecurity is uniquely difficult because bioattacks and biodefenses scale asymmetrically. There is a massive offensive advantage: compared with defenses such as pathogen-agnostic detection and medical countermeasures, attacks take less time to deploy, cost less, and benefit more from advances in AI. Broadly, attacks are relatively more bottlenecked by information and tacit knowledge, while defenses are relatively more bottlenecked by physical properties, such as manufacturing capacity and physical infrastructure. By removing friction in accessing information, including tacit knowledge, AI is poised to accelerate attackers’ progress more than defenders’.

The US should therefore build a strong defense to prevent, detect, and defend against biological threats. We recommend:

• Prevent dangerous biological threats from being created and released: Congress should secure the DNA supply chain, such as by passing Biosecurity Modernization and Innovation as part of the American Biotechnology Competitiveness Act, requiring DNA synthesis providers to screen customers and orders while minimizing burdens on legitimate research. Congress should also equip CAISI to evaluate new models for biological risks and to develop safeguards standards to reduce dangerous uplift without degrading legitimate use. Finally, OSTP should coordinate NIH and NIST pilots of tiered access controls for narrow subsets of pathogen data most likely to enable misuse and provide the gene synthesis screening framework requested in Executive Order 14292.

• Detect threats before they spread widely: The CDC should invest $80M annually in a federal pathogen early-warning system, adding metagenomic sequencing to existing traveler, wastewater, and clinical surveillance networks, so the US can rapidly detect novel biological threats. Using the resulting information, the Defense Innovation Unit should continue to fund the development of better AI tools capable of scanning these billions of data points for emerging threats, and explore new options for better biosurveillance.

• Defend the population and shorten the path out of the pandemic: Congress should sustainably fund the Strategic National Stockpile to grow and maintain day-zero reserves of respirators and medical countermeasures for future pandemics. Congress should also sustainably fund warm-base manufacturing capacity to ensure that we are prepared to rapidly manufacture countermeasures for novel threats. The Biomedical Advanced Research and Development Authority (BARDA) should provide advance purchase commitments for broad-spectrum antivirals. The Department of Health and Human Services should fund large-scale trials of air-cleaning technologies, such as Far-UVC and glycol vapors, and recommend evidence-based updates to ASHRAE Standard 241. Finally, Congress should provide incentives for buildings to procure these air-cleaning technologies.

Many more projects should be undertaken in biosecurity, as well as in other critical areas. This list is far from exhaustive. IFP is currently developing more concrete project proposals to increase AI resilience as part of The Launch Sequence.

The most significant weakness in the piece is the false assumption that because it is difficult to design and manufacture a bioweapon with current technology, it will remain so. This particular opinion is actually highly pessimistic about the rate of growth of technology. Many things used to be difficult that are no longer so. They become accessible as technology advances.

Synthetic biology is not an exception to this, as we have watched costs fall and accessibility rise without fail since its inception. Before 2012, editing a specific site in a genome meant using zinc finger nucleases: custom-engineered proteins that cost $5,000+ to order, and were never widely adopted because they were so difficult to engineer well. CRISPR replaced this. You order the RNA guide sequence and buy the rest off the shelf, with total cost being as low as $30. This is a strong example of an engineering capability that was once siloed to a handful of specialist labs and is now an undergraduate exercise. Synthesis costs have also plummeted. Synthesizing the poliovirus genome cost about $300,000 in 2002, and the materials would cost under $1,000 today.

AI is now an ordinary tool for synthetic biology. DeepMind’s AlphaFold has mostly solved protein structure prediction, a problem which was once thought to be intractable for computers. Frontier AI models have safeguards because frontier AI labs are concerned about their own models’ capabilities. Some may argue that talking about safety is just for good marketing, but frontier labs are taking actions that hurt their bottom line. Anthropic has gone so far as to make Fable unusable for even basic biology. This would be nonsensical if their concerns were unfounded. Open-source models with lower safeguards continue to advance just a few steps behind. When the enabling technology is improving at such a high speed, I find it surprising to hear people claim that the fact that something in biology is difficult now means it will continue to be so this next decade.

This outlook is striking because Olvera’s blog is called “Positive Sum.” She seems to write optimistically about progress, taking the position that it is institutional friction and regulation preventing everything from becoming cheap and abundant. I agree with this, and to me it stands in contrast to this piece. Everything will get better over time, except the technology to make bioweapons?

She also focuses on lone actors and smaller groups, without acknowledging the likelihood of effective state bioweapons programs. We know that multiple states have active bioweapons programs. As I wrote in a prior post, “the most recent 2025 report from the Department of State affirmed that Russia and North Korea maintain offensive biological weapons programs. It raised continued compliance concerns re: Iran and China, noting that PLA military medical institutions are conducting research with potential bioweapons applications, and warned (for the first time) that China is capable of using publicly available AI/ML tools to advance efforts related to bioweapons.” States already possess both BSL-3 and BSL-4 labs, as well as accompanying trained personnel.

She says of potential adversaries: “For almost any goal they have, a bomb, a gun, a chemical, or a cyberattack is cheaper, faster, and more controllable.” However, the “almost” is critical here. There are several reasons why a threat actor would choose to deploy a bioweapon over other options, not the least of which are their capability for unprecedented, self-propagating destruction and the difficulty of attributing an attack to a particular actor.

She lists five main reasons for bioweapons not being the weapon of choice, which I will address one by one:

1. “You have to be comfortable killing your own people. Pathogens kill indiscriminately. Few groups are this malicious and extreme. Groups that are tend toward paranoia and rigid hierarchies that make them less effective.”

The two actors we are most worried about, state actors and terrorists, overcome this objection. State actors can stockpile vaccines for their own population and vaccinate their scientific personnel –– the Soviets did both during the Cold War. Terrorists may view loss of life on their side as a form of martyrdom, or even total loss of life as the goal, like Aum Shinrikyo.

2. “You need significant resources. Bioweapons capable of harming many people need equipment, chemicals, and professionals trained in different fields.”

First, state actors have significant resources. Second, the resources you need to do this are falling year-over-year. Third, we have already seen cult Aum Shinrikyo recruit scientists and purchase equipment, so we know it is possible. Last, many terrorist groups are in fact quite well-funded. Expecting capital to be the barrier, as costs rapidly fall, does not make sense.

3. “You need an institution for access. Many items are too niche for black markets. Essential equipment like biosafety cabinets or fume hoods requires professional installation and ongoing manufacturer support. Some materials are restricted; for example, viruses need host cells, cell lines are sold only to institutions, and plasmids require institutional credentials. In the United States, only tens of thousands of people have sustained access to such facilities and supplies.”

You do not need an institution for most specialized equipment and materials. Most things can be found in an average biology lab or acquired used online. A determined actor can find workarounds for any part of the workflow, including finding providers who don’t screen nucleic acid orders, routing orders through proxies, or committing theft. We should also remember that a state actor faces none of these constraints, and that a threat actor could be an insider at an existing institution. The FBI concluded that the 2001 anthrax letters came from a government researcher with institutional access. “Tens of thousands of people” is a lot of people.

4. “Your team must be both highly competent and completely secret. A contamination incident could destroy the pathogen or infect the team.”

The Soviets ran an effective 30,000-person bioweapons program for two decades. 30,000. The program, Biopreparat, had a civilian cover. It did not completely escape detection. Western agencies had some suspicions, and the 1979 Sverdlovsk anthrax release was noticed internationally but mostly successfully blamed on contaminated meat. However, the existence of a program of that extent was only confirmed when Vladimir Pasechnik and Ken Alibek defected. There were multiple instances of contamination and infection of the scientists who worked on the program, including Alibek himself, and the program continued. One researcher, Nikolai Ustinov, died from contamination with Marburg. His colleagues harvested the mutated virus from his body, found it more virulent, named it Variant U, and weaponized it.

5. “You need to convince and retain experts from different domains. e.g. biology, chemistry, engineering, aerosolization, or fieldcraft to carry out attacks. Recruiting without detection is hard too.”

Even non-state actors are not limited to poorly trained or low-competence individuals, yet people repeatedly make this assumption. It’s rare, but history shows us clear examples of highly educated and technically capable people who chose to pursue sophisticated terrorism. Consider Ted Kaczynski, a Harvard-trained mathematician who taught at Berkeley before becoming known as the Unabomber, and Ayman al-Zawahiri, a trained surgeon who ran al-Qaeda’s biological weapons effort. Al-Zawahiri recruited Yazid Sufaat, a U.S.-trained biochemist, to do some of al-Qaeda’s benchwork.

She closes this section with Aum Shinrikyo as an example of why bioweapons are hard: “Aum Shinrikyo, a Japanese cult with ~$1 billion, failed in all ten of their attempts to release biological agents.” First, I want to note that Aum’s bioweapons budget was actually something like only $3M/year out of the ~$1 billion. Second, I agree that bioweapons are hard, and that Aum is evidence for this. However, I think Aum is even better evidence that a group has already had the motivation, capital, equipment, and staff to commit a biological attack. In modern times, such a group would have access to significantly better information and technology.

Regarding laboratory benchwork, the piece also overstates how difficult basic techniques are. Olvera cites pipetting as an example. I have spent many hours in the lab pipetting small volumes. It took me ~1 week to get good at it, and the same is true for the dozens of other students I worked with. The idea that benchwork represents some insurmountable barrier is false. I would argue that benchwork is not hard because of technique; it is hard because biology is unpredictable. Sometimes your antibodies just don’t work, etc. This would not be significantly more difficult for a threat actor than for a newer scientist. We should also not assume this threat actor will have no access to training, nor that they could not be a scientist themselves. Even with just a few months of training, there is not a lab technique I can think of that is out of reach. Moreover, soon more cloud labs like Emerald will come online, robotic pipettes and cell culture will become more common, and labs will shift, as all sectors always have, towards increased automation. Soon, a threat actor need not even be in the lab to make a weapon.

Olvera points out correctly that AI uplift for benchwork is not currently a major concern. She writes, “The largest such study to date, by ActiveSite, indicated that non-experts with access to mid-2025 LLMs performed better on individual virology tasks but saw no meaningful improvement in end-to-end workflows. Success rates on these basic workflows remained below 8%.” Though the measurable uplift for non-experts is small, meaningful improvement on individual tasks eventually leads to improvement in end-to-end workflows, even if it does not now. Let’s do the math. Assume we have five sequential steps at 60%. I chose 60% because this would yield a 7.8% success rate, close to the sub-8% figure the study reports. If you got each step to 80%, the same workflow would make a big jump to a 33% success rate. AI is highly capable of benchwork troubleshooting, as indicated by SecureBio’s Virology Capabilities Test, which demonstrated that o3 performed better than 94% of expert virologists, even in April 2025. This is evidence that AI will be able to provide uplift to some of the steps, especially as it improves. The ActiveSite study was also a time-restricted study. In biology, it takes time to get every step right. This means that things go from not working at all to working well, very quickly. A study like this run over a year, I believe, would have substantially different outcomes. And that is the experiment we are running in the real world right now.

The claims Olvera makes that I disagree with most are: “Any pathogen that a team could plausibly create is one that already exists in nature. An engineered version of COVID-19 would have about the same effect as an infected person coughing in a crowded room.”

First, progress in generative biology undermines the claim that any plausible engineered pathogen would be something that already exists in nature. AI model Evo has already successfully designed novel viral genomes and the Arc Institute demonstrated them as functional in the lab. She argues that this particular example is irrelevant: “Some people cite bacteriophages as evidence of AI-created novel viruses; however, practitioners don’t consider them truly novel. These are 7% different from nature, which is seen as normal evolutionary change.” 7% different is meaningfully different, enough to be considered a new species under some taxonomies. For instance, under criteria from the International Committee on Taxonomy of Viruses, a phage genome below 95% ANI to any known relative qualifies as a new species. Evo’s most divergent viable design from this trial, at 93.0% ANI, qualifies. Calling this normal evolutionary change is only true in the sense that speciation is normal. “Normal evolutionary change” at the pace of a lab and not at the pace of evolution is decidedly not normal.

Olvera then writes, “These engineered bacteriophages don’t need to evade the immune responses that even bacteria possess, making it unclear whether they’re viable compared to natural viruses that have survived that selection pressure.” The point of the Arc Institute project was not to make more infectious bacteriophages, just novel ones. They made them intentionally similar to ΦX174, so this is not relevant to the issue at hand. Even if this were not considered a “novel virus,” technology is clearly progressing towards this. One can argue about the timeline, but we will be able to do this. We have seen attempts to make chimeric viruses even without the aid of AI. The design space is demonstrably larger than simple modifications of existing viruses.

Second, the claim that “an engineered version of COVID-19 would have about the same effect as an infected person coughing in a crowded room” is dependent on the type of engineering. I think Olvera is trying to say that it would just make COVID more transmissible, glossing over the fact that even if that were all the engineering was able to accomplish, this would still have resulted in millions more deaths. We have in some sense already run the transmissibility experiment: Alpha and Delta spread faster than the original strain, and each drove a wave of deaths far larger than the original strain would have caused. While I think significant changes in lethality and transmissibility are possible, even a modest change to one parameter does not mean a modest outcome.

Olvera adds to the argument above by pointing out that releasing 1918 influenza today would not be “1918-bad,” because we have cross-immunity as most immune systems recognize flu descendants and we have antivirals and antibiotics for the secondary pneumonia that killed most victims. This is of course true, especially since we also have much better medical infrastructure and medicines now. The point I want to make in response is that cross-immunity is precisely the kind of property an engineered modification would target. (Also, the antibiotic argument assumes that we have some very solid supply chains.)

Olvera also writes, “Engineering a virus that is both extremely lethal and highly contagious, worse than anything nature has produced, would be bad. But many biologists doubt this is possible even with perfect technology.” However, there is very meaningful disagreement among experts on how difficult it would be to engineer something both highly lethal and highly transmissible. Only mentioning that many biologists consider this impossible even with advanced technology does not at all reflect the full range of assessments in the field. The lethality versus transmissibility tradeoff is just an observation about what natural selection optimizes for. Engineering can bypass the tradeoff, and some natural viruses already do. Smallpox has an R₀ of ~5 with a fatality rate near 30%. HIV is ~100% fatal untreated, and transmits for years with an R₀ of ~2-5. (For comparison, the original COVID strain had an R₀ of ~3 and an infection fatality rate of ~0.5%.) Both bypass the tradeoff by separating the transmission phase from the lethal phase, a technique which could also be engineered into a weapon.

Olvera states that she interviewed dozens of biosecurity experts and lab practitioners, but she does not name any of them (except for Michael Montague) or describe their specific backgrounds or affiliations. This makes it difficult for me to assess how representative their views are, but my conversations with dozens of biosecurity experts and lab practitioners, as someone who has worked both in biosecurity policy and in a lab, have been very different.

Olvera closes by explaining why her piece is not conventional wisdom. She reminds us that often null results don’t get published, and advocacy groups need urgency. You could cynically say that biosecurity people are motivated to keep their jobs. I am sure that some people are. You should also consider the more likely case that most of us are motivated because we believe the threat is real and significant. Speaking as someone who only recently made biosecurity their full-time priority, I can assure the reader that I would love to be working on something else, and for this not to be such an urgent problem. This is also the sentiment of many people I know in the field. I understand this is anecdotal evidence, but so is her claim that “many leaders in biosecurity and national security organizations” thanked her privately for writing this conclusion.

Some claim that AI helps defenders too, and there is a real argument that in cybersecurity AI improvements can favor defense. Unlike cybersecurity, biosecurity has a profound offense-defense asymmetry. An attacker does not have to go through clinical trials or pay for manufacturing. An attacker has time to design a weapon and can choose when to release, but the defender can mostly only start responding after the attack.

At the end of the day, pieces like this encourage complacency. It is unclear exactly how quickly technology will advance over the next few years, and state actors are already capable of massive harm. Complacency is costly: bioweapons can be weapons of unprecedented mass destruction. We should treat biosecurity as urgent.

Disclaimer: This piece represents my personal views and my personal views only. It does not represent the opinion of any organization I work for or with.

I think there are actually two axes of belief here:

1. Your belief in the potential of AI as unprecedentedly, exponentially capable.

2. Your degree of optimism in people and Good.

By the second axis, I do not mean thinking that all people are good and therefore no one will misuse AI. To believe a bad actor could turn AI into a catastrophe is to credit human beings with staggering capability –– the same bet as believing a defender could stop them. You do not have to think humanity is wicked to be a “doomer,” or even that AI misalignment is a strong risk. You only have to think that, out of eight billion people, one is enough.

This second axis is therefore really measuring belief in good actors’ capabilities, which is, in some sense, one’s belief in Good triumphing broadly. This is belief in the hero’s camp: that Good actors, be they human or AI, can triumph in the face of anything.

Claude has done me the honors of plotting what I mean here:

One is typically labeled a “doomer” precisely because they are actually optimistic about the rate of growth of AI technology and have realized the dramatic implications of such a thing. One can be labeled an “optimist” simply because they don’t think AI can ever really achieve “superintelligence,” that it is simply normal technology. These people are actually pessimistic about the rate of growth of technology.

Aforementioned “doomers” on the right side of the graph split into multiple camps along the Y axis, like those who think we can and should stop technology development (or that the world will end regardless) on the bottom right, and those who believe we should accelerate it to defend against it on the top right. I have labelled the upper right after Clifford’s defense acceleration, or “def/acc,” concept. It is a label precisely for those of us who believe technology can protect us from the risk technology creates.

The “normal tech” people on the left split into multiple camps as well, like those who believe that we can always stay ahead of AI because it’s just a tool on the top left, and those who believe it is overhyped on the bottom left. You will notice that it is harder to distinguish between camps on the left. When you believe the technology is bounded, it doesn’t really matter what you think about the human condition or the triumph of Good. If I had to, I would split them between the Andreessen-esque “Techno-optimists” at the top, many of whom also fall past the y-axis into the defense-accelerationist camp, and the folks at home who haven’t really tried AI for advanced work or followed the news at the bottom. This group on the bottom left is most people. It also includes people in denial and people who don’t like any technology at all. You (usually) have to be open minded about technology to appreciate its potential.

There are also members in both bottom quadrants of the graph who are deeply pessimistic about future society, regardless of how powerful AI technology becomes. Since it is likely already good enough to put many people out of their jobs, there are people in these quadrants who do not believe there can be a good future, period, without massive social change. The opposite feeling is reflected by some folks in the upper two quadrants –– those who believe that things will be good no matter what because they just don’t really believe anything bad could happen.

The group on the bottom right contains the Yudkowsky camp, who believe that if anyone builds superintelligent AI, everyone dies.

We must remember that many “doomers” of the right side of the graph, especially of the Yudkowsky bottom right flavor, are coming from the Bostromian perspective of preventing astronomical waste. They are so optimistic about technological progress that they believe a future of nearly infinite wonder and qualia is possible through it. To them, the primary opportunity cost of catastrophe is foreclosing technological progress toward that future. It would be difficult to be more optimistic about technology.

The dominant group in the upper right quadrant are the double optimists. We believe that the technology has near-infinite potential, and that mankind can win.

We agree with the Yudkowsky camp about technological potential. We disagree that good technology, built by good people, can’t triumph now.

It may be unrealistic to be this hopeful. The Yudkowsky camp could be right. There are asymmetries that will be hard to overcome between offense and defense in many threat models, especially in biosecurity. Some in our quadrant are optimistic about defense capability more basically; however, given those asymmetries, I think that is a harder claim to defend. Others would prefer to slow frontier AI development, and choose to accelerate defense only because this seems unlikely to happen. Their ideal is probably closer to differential development.

The part of the quadrant of double optimism that I live in is almost a kind of faith. It is faith in people and Goodness, and faith in unbounded technological progress. We do not expect to win because the odds are good or the threat is small. We believe in human agency, and maybe even in the agency of a “Good” AI. This is Dario Amodei’s camp.

It is easy to call the people shouting out about the potential risks of AI “doomers.” People still remember those who yelled out about the dangers of the industrial revolution, and even the printing press. What people seem to forget is the coalition of people who shouted out and worked tirelessly to develop frameworks for atomic weapons strategy after the development of the atomic bomb. (If it were possible to shield against nuclear weapons, there would have been a def/acc camp back then. They did try.)

The point of all this is that we should stop calling people warning about and then actually working on serious risks doomers; they aren’t. We are optimists, and defense accelerationists. We should stop calling people who think AI is a normal tool optimists; they aren’t. They are capability pessimists.

When AI enables us to make any technology or business with a dream and a keystroke, it will be because our cyber systems were protected by people working on cybersecurity.

When AI enables a world free of disease, it will be because people working on biosecurity prevented AI-enabled bioweapons from destroying civilization.

When AI enables us to colonize the stars, it will be because AI researchers worked for decades to create AI that aligns with mankind.

Def/acc.

Further reading/references:

AI as Normal Technology — Arvind Narayanan and Sayash Kapoor

The Techno-Optimist Manifesto — Marc Andreessen

My techno-optimism — Vitalik Buterin

Introducing def/acc at EF — Matt Clifford

Notes on Differential Technological Development — Michael Nielsen

Astronomical Waste — Nick Bostrom

If Anyone Builds It, Everyone Dies — Eliezer Yudkowsky and Nate Soares

Machines of Loving Grace — Dario Amodei

P.S. If you were curious about where some of the models place themselves on my axes when prompted, I have added that below. Please comment where you would place yourself!

Disclaimer: This piece represents my personal views and my personal views only. It does not represent the opinion of any organization I work for or with.

Last year, Arvind Narayanan and Sayash Kapoor published a now well-circulated essay, AI as Normal Technology. The essay is still popular, I think, primarily because people would like it to be true, myself included. It is a terrifying proposition to acknowledge how different AI may be from “normal” technology. However, acknowledge it we must –– the urgency of developing proper governance and technical progress on alignment cannot be overstated.

I think the essay has significant flaws. This response walks through the most important ones in an effort to demonstrate the importance of acknowledging the tremendous nature of the changes superintelligent AI may bring.

The essay consists of Narayanan and Kapoor laying out a scenario they deem to be the median outcome, in four parts, roughly as follows:

Part I: Speed. They argue that transformative economic and societal impacts will be slow and there is a distinction between AI methods, AI applications, and AI adoption, including different timescales.

Part II: Division of labor. They discuss potential division of labor in a manner they say does not apply to “superintelligent” AI, which they “view as incoherent as usually conceptualized.” They argue control is primarily in the hands of people and organizations; indeed, a greater and greater proportion of what people do in their jobs will be AI control.

Part III: AI risks. They argue viewing AI as normal technology leads to fundamentally different conclusions about mitigations compared to viewing AI as being humanlike.

Part IV: AI policy. They argue we should reduce uncertainty as a first-rate policy goal, use “resilience” as the overarching approach to catastrophic risk, and that drastic interventions premised on the difficulty of controlling superintelligent AI will make things worse if AI turns out to be normal technology.

If the question is “will the labor market be unrecognizable in 2035,” they’re probably closer to right than wrong. However, I do not think this is the question. The question is, “might this be more transformative for what it means to be human than any other technology in the history of man?” I think the answer to that question is yes. If so, it would be quite difficult to settle on thinking that AI is “normal technology” in the sense that Narayanan and Kapoor are calling it so.

In what sense are they calling it so? Here are some bits of the essay I think exemplify what they mean by normal technology; I have bolded the parts I disagree with at this time:

To view AI as normal is not to understate its impact—even transformative, general-purpose technologies such as electricity and the internet are “normal” in our conception. But it is in contrast to both utopian and dystopian visions of the future of AI which have a common tendency to treat it akin to a separate species, a highly autonomous, potentially superintelligent entity.

The statement “AI is normal technology” is three things: a description of current AI, a prediction about the foreseeable future of AI, and a prescription about how we should treat it. We view AI as a tool that we can and should remain in control of, and we argue that this goal does not require drastic policy interventions or technical breakthroughs. We do not think that viewing AI as a humanlike intelligence is currently accurate or useful for understanding its societal impacts, nor is it likely to be in our vision of the future.

The normal technology frame is about the relationship between technology and society. It rejects technological determinism, especially the notion of AI itself as an agent in determining its future. It is guided by lessons from past technological revolutions, such as the slow and uncertain nature of technology adoption and diffusion. It also emphasizes continuity between the past and the future trajectory of AI in terms of societal impact and the role of institutions in shaping this trajectory.

I appreciate that Narayanan and Kapoor think they are not understating its impact, but they go on to make arguments that explicitly understate its impact.

For example, they make good points about things like benchmarks not always measuring real-world utility, there being a gap between capability and reliability, and diffusion, especially in safety-critical domains, being slow and constrained by human organization and institutional change. These points are made with the implicit intention of minimizing the impact of AI.

They also say that the essay has “the unusual goal of stating a worldview rather than defending a proposition,” then go on to defend their propositions, not against any possible counterargument, but certainly to justify them against the prevailing narrative.

I believe their view of AI as merely a new tool, particularly one that does not require technical breakthroughs to control, is deeply wrong. I think their policy conclusions may be correct, but for reasons that have less to do with thinking AI is normal technology and more to do with thinking it isn’t. Because AI is not normal, it is especially true that we must not decelerate our defensive capabilities as our adversaries accelerate their offensive ones. Because AI is not normal, it is more critical than ever that the United States, and the West broadly, maintains control over the direction of this technology.

On Diffusion and Reference Class

To reach their conclusion, Narayanan and Kapoor’s central logic is jumping from something like “impacts arrive gradually” to “this is continuous with past technologies” to “therefore AI is normal technology.” Impacts may indeed arrive gradually –– at first. It was predictable that something like this would be developed, but it is a non sequitur to conclude that this makes AI “continuous” and therefore “normal” technology. Something being predictable does not make it continuous. I believe these are distinct claims.

AI being “not normal” does not require Bostromian fast takeoff. The transformation can be slow and still be categorically different from prior technological transitions.

Most of their arguments in this section rely on somewhat twisted choices of relative measure.

They write:

Even outside of safety-critical areas, AI adoption is slower than popular accounts would suggest. For example, a study made headlines due to the finding that, in August 2024, 40% of U.S. adults used generative AI.13 But, because most people used it infrequently, this only translated to 0.5%-3.5% of work hours (and a 0.125-0.875 percentage point increase in labor productivity).

It is not even clear if the speed of diffusion is greater today compared to the past. The aforementioned study reported that generative AI adoption in the U.S. has been faster than personal computer (PC) adoption, with 40% of U.S. adults adopting generative AI within two years of the first mass-market product release compared to 20 % within three years for PCs. But this comparison does not account for differences in the intensity of adoption (the number of hours of use) or the high cost of buying a PC compared to accessing generative AI. Depending on how we measure adoption, it is quite possible that the adoption of generative AI has been much slower than PC adoption.”

The claim that the speed of technology adoption is not necessarily increasing may seem surprising (or even obviously wrong) given that digital technology can reach billions of devices at once. But it is important to remember that adoption is about software use, not availability. Even if a new AI-based product is instantly released online for anyone to use for free, it takes time to for people to change their workflows and habits to take advantage of the benefits of the new product and to learn to avoid the risks.

Thus, the speed of diffusion is inherently limited by the speed at which not only individuals, but also organizations and institutions, can adapt to technology. This is a trend that we have also seen for past general-purpose technologies: Diffusion occurs over decades, not years.

The PC comparison is not appropriate here. AI is significantly more comparable to social media, search, or the smartphone, all of which took years rather than decades to “takeoff.” They say it is about software use as opposed to availability, but use a survey-based measure of adult adoption which excludes API/enterprise use and likely misses the heaviest individual users.

They then discuss something called the “bitter lesson,” a term that comes from a 2019 essay from Richard Sutton. Narayanan and Kapoor write:

The “bitter lesson” in AI is that general methods that leverage increases in computational power eventually surpass methods that utilize human domain knowledge by a large margin. This is a valuable observation about methods, but it is often misinterpreted to encompass application development. In the context of AI-based product development, the bitter lesson has never been even close to true.²³ Consider recommender systems on social media: They are powered by (increasingly general) machine learning models, but this has not obviated the need for manual coding of the business logic, the frontend, and other components which, together, can comprise on the order of a million lines of code.

As they mention, the intelligent part of recommenders has followed along Sutton’s predicted path: content-based filtering or neighborhood-based collaborative filtering → matrix factorization → deep learning → transformer-based models. Each transition replaced more human-led work with general model-led methods, scaled with more compute.

The essay they cite is their own, which argues that turning LLMs into reliable products requires substantial engineering work that companies underestimated, but does not support the strength of claim they are making. In the year since this essay was published, I believe these statements have already been falsified. We are absolutely moving towards more model-led business logic, frontend, and other components. Some companies now write minimal code by hand, with software engineers preferring to use Claude Code, Codex, or Cursor. Claude Code itself is about 90% written by Claude Code. I have also seen multiple business briefs written entirely by AI, and have little doubt decisions at major businesses are now being discussed with and possibly deferred to AI.

Their arguments about slow absorption of institutions are fine; however, slowness of institutional adoption tells you about institutions, not about the thing being adopted. And it certainly does not tell you about what things will look like 50 years from now. Basically, they might be completely right about diffusion, but this is not relevant to normalcy.

They write that “Even if every task that humans do today might be automated one day, this does not mean that human labor will be superfluous.” Generations that remember other automations think this way, and I understand why. They remember other groups of people speaking out regarding past advancements in technology. It has previously been correct to believe this. To those folks, I ask you to extrapolate further –– are we really to believe there are infinite jobs to regress to if energy is cheap, AI can replace human intelligence, and AI-enabled humanoid robots can replicate all human labor? What is left?

There are many arguments against this, the best of which is that this will lead to some kind of post-commodity economy. Economics Professor Alex Imas writes about this in his article, What will be scarce? He says there is a world “where a growing share of expenditure goes toward goods and services whose value is inseparable from the human who provided them.” I agree with him on some level, as well as others who have made this point, like Seb Krier, Google DeepMind’s policy lead. However, even in the best of cases, this is a disaster for the developing world, and it is unclear that AI will not simply create a better alternative to the goods and services that typically fall into this category or that tastes will not change. People may eventually not care about whether Starbucks handwrites your name on a cup, or there may be a robot that engraves it with your own personal portrait that people want more. The television has largely replaced bards, and the market ratio of TV & movies to theater is at least 20:1. (I hold this view very weakly relative to the rest of this response.) Essentially: slow as it may begin, diffusion will happen, and it is unclear what jobs survive this.

The deeper issue is their choice of reference class. Narayanan and Kapoor bring up electricity as an analogous thing that took time to reshape the economy. This may very well be a solid argument that general-purpose technologies take time to reshape economies. It is unclear to me that AI belongs in the reference class of electricity rather than in some new category. General-purpose though it may be, commodification of human thought, creativity, and (soon) physical labor, is normal until it isn’t.

In fact, everything never changes, until it does. The argument from continuity for AI being normal has a structural problem in generalizing from the fact that many past predictions of discontinuity were wrong. This reasoning, taken to its limit, says nothing can ever be a change greater than any that has come before.

I believe this is more of the Lucretius problem. People are biased (usually rightly so) by the magnitude of changes they have seen before.

A little river seems to him, who has never seen a larger river, a mighty stream; and so with other things –– a tree, a man –– anything appears greatest to him that never knew a greater. –De Rerum Natura, Titus Lucretius Carus

As Narayanan and Kapoor acknowledge on some level, step changes are normal. Somewhere in the history of a species, there are technologies that change what the species is. I think that AI is one of these great step changes.

Every prior technology has complemented cognition. This one emulates it, and enables the simulation of essentially any kind of labor. Every prior general-purpose technology in history improved human capability by giving us better tools to act on the world with our minds. AI is the first technology that acts on the world after being prompted simply with our faintest desires. It is the first technology that can act on the world with its mind. It is the first and only technology which in theory could continue to expand and consume energy on its own indefinitely.

On Intelligence

One of Narayanan and Kapoor’s other lines of argument is to deny that something like “superintelligence” is coherent or could exist by denying that such a thing as intelligence exists. It is a bizarre line of reasoning to follow:

Can AI exceed human intelligence and, if so, by how much? According to a popular argument, unfathomably so. This is often depicted by comparing different species along a spectrum of intelligence.

However, there are conceptual and logical flaws with this picture. On a conceptual level, intelligence—especially as a comparison between different species—is not well defined, let alone measurable on a one-dimensional scale.

They also include this chart, which they say is wrong:

I cannot fathom that someone would try to deny that intelligence between species varies measurably. They are trying to pretend AI is normal by saying the thing that it is is not measurable, but intelligence is measurable. You, reading this, are smarter than a mouse. So is Opus 4.7. Intelligence being multidimensional does not mean it is not orderable, nor does it mean that those dimensions don’t correlate/are not reducible.

I’ve many times heard the argument that AI cannot be smarter than the smartest human because it only has human data to train from, or that it can never do novel work for the same reason. I think it should be obvious that this is not true. First, it doesn’t only have human data to train from –– soon we will have AIs trained on any real-world data they can collect. Second, even if for some extraordinary reason you were constrained by human intelligence, I implore you to imagine one million von Neumanns collaborating on problems together at machine speed. Lastly, there is mounting evidence emerging for technical creativity and ability to do novel work. This includes novel mathematical proofs, especially given the recent announcement from OpenAI on one of their models disproving a central conjecture in discrete geometry. We are watching these assumptions fail in real time.

Narayanan and Kapoor then go on to say that actually, intelligence doesn’t even matter:

More importantly, intelligence is not the property at stake for analyzing AI’s impacts. Rather, what is at stake is power—the ability to modify one’s environment.

To which I say: intelligence historically has always been correlated with the power to modify one’s environment. AI podcaster and writer Dwarkesh Patel has an interesting piece countering this, called The mistake of conflating intelligence and power. His argument is that the world’s most powerful people aren’t its smartest, which is fair. However, the most powerful humans do “max out” intelligence in the only way they can, since you can’t really raise your own: they hire all the physicists. Essentially, intelligence is necessary but not sufficient for power. You also need courage and goals. Computers don’t need courage, will be given goals by default, and can hire both people and other AIs.

Narayanan and Kapoor add the image below to illustrate how they conceive of technology, not intelligence, being the source of capability. I would be curious to understand how they think we developed technology in the first place, if not from intelligence.

They then say:

De-emphasizing intelligence is not just a rhetorical move: We do not think there is a useful sense of the term ‘intelligence’ in which AI is more intelligent than people acting with the help of AI. Human intelligence is special due to our ability to use tools and to subsume other intelligences into our own, and cannot be coherently placed on a spectrum of intelligence.

This is approximately true by construction and not true in any real sense of the word. They’ve defined the relevant unit as “human + AI,” so any AI improvement gets absorbed into the unit and counted for the human.

In reality: AIs can already use tools. AIs will be able to “subsume” other intelligences to their own, more literally than a person is able to do so. AI was trained by subsuming our intelligence. It may lack agency in some sense, but this doesn’t make it any less dangerous. It will be able to decide, potentially with little to no human input, what it thinks needs to be done.

We have seen Narayanan and Kapoor’s argument before. For decades, top chess players and critics argued that computers would never play chess at the level of the best humans, because human intelligence is “special”.

IM David Levy bet AI researchers in 1968 that no computer would beat him within ten years, and won the bet in 1978. In 1996, Levy predicted Kasparov would beat Deep Blue six games to zero, and said, “I’d stake my life on it.” Kasparov had played 32 chess computers simultaneously in 1985 and beat all of them. He rebuffed IBM’s offer to split the Deep Blue prize money 60-40 in 1996. Karsparov won the match 4-2, and the rematch was set for 1997. Of his attitude at the rematch, he said he thought, “I will beat the machine, whatever happens. Look at Game One. It’s just a machine. Machines are stupid.” He lost the rematch.

When Deep Blue won, the response was not to update the understanding of how good machines are, but to decide chess was “just brute force,” so for a computer to be real intelligence it had to learn Go or language, or something. The goalposts moved, and continue moving to this day. Kasparov’s response was to develop “centaur chess” i.e. human plus computer playing as a team and to argue, from 1998 through at least 2017, that centaur teams beat the strongest computer alone. In truth, sometime around 2013 the human became useless in chess, and the gap has only widened since.

Narayanan and Kapoor are making the same two arguments, applied to AI generally. They claim intelligence isn’t well-defined enough for modern AI to count, shifting the goalpost. They claim human + AI is the relevant unit, that there is no useful sense in which AI is more intelligent than people acting with AI. There is no reason to think the argument that didn’t work for chess models will work for language models.

They know the centaur story, and they think it is a question of speed. They write:

Human abilities definitely have some important limitations, notably speed. This is why machines dramatically outperform humans in domains like chess and, in a human+AI team, the human can hardly do better than simply deferring to AI…In many other areas, including some that are associated with prominent hopes and fears about AI performance, we think there is a high ‘irreducible error’—unavoidable error due to the inherent stochasticity of the phenomenon—and human performance is essentially near that limit

To this, I point out that protein structure prediction isn’t speed-bound. Humans plateaued at ~40% accuracy and AlphaFold reached ~90%. Theorem proving isn’t speed-bound, and recent systems are producing novel proofs for problems that humans have spent years working through. In both cases, humans are/were not near an “irreducible” limit, just a human one. I do not think we can tell how stochastic the world is from our position of lesser understanding. Many things may be more solvable than we think, and AI may be able to do it.

On Speculative Risks, Control, and Access

Narayanan and Kapoor are not worried about catastrophic risk, which they describe as speculative:

In the view of AI as normal technology, catastrophic misalignment is (by far) the most speculative of the risks that we discuss. But what is a speculative risk—aren’t all risks speculative? The difference comes down to the two types of uncertainty, and the correspondingly different interpretations of probability.

In early 2025, when astronomers assessed that the asteroid YR4 had about a 2% probability of impact with the earth in 2032, the probability reflected uncertainty in measurement. The actual odds of impact (absent intervention) in such scenarios are either 0% or 100%. Further measurements resolved this “epistemic” uncertainty in the case of YR4. Conversely, when an analyst predicts that the risk of nuclear war in the next decade is (say) 10%, the number largely reflects ‘stochastic’ uncertainty arising from the unknowability of how the future will unfold, and is relatively unlikely to be resolved by further observations.

By speculative risks, we mean those for which there is epistemic uncertainty about whether or not the true risk is zero—uncertainty that can potentially be resolved through further observations or research.

If we are to grant their distinction here between what they are calling stochastic uncertainty (nuclear war scenario) and what they are calling epistemic uncertainty (asteroid scenario), it is clear the uncertainty about AI is closer to the nuclear war scenario than to the asteroid scenario. AI is currently rather representative of stochastic unknowability. The asteroid case is a single physical trajectory resolvable by one better measurement and AI risk is the opposite. It is not a question of “what is the true underlying probability our measurements just can’t see right now?” but “how will billions of deployment choices, inevitable adversarial use, and emergent system behaviors unfold?”

Moreover, just because a risk is low, which is what they really seem to mean here, does not mean it should not be prioritized. The expected value of addressing a risk is not based simply on its probability, but on probability*impact. The type of risk we are discussing is so astronomical that the impact dwarfs the probability in scale.

I also reject their categorization of catastrophic risks as all rogue-unaligned-AI flavored. The most concrete catastrophic risk is biosecurity, and it does not require AI to act autonomously. It only requires AI to lower the technical floor for designing and synthesizing pathogens, and frontier models already provide meaningful uplift. It would not be normal for anyone with a computer to have the ability to engineer a weapon of mass destruction more lethal than any nuclear bomb. Even if this technology is safeguarded well enough to keep it out of the hands of most bad actors, and no one releases highly capable open source models, it will be available to nation states. AI-enabled, bioengineered pathogens are coming, and this essay does nothing to address how significantly different the world is in which motivated actors can produce catastrophic harm.

That being said, rogue-unaligned-AI flavored risks are less theoretical than they argue. Even if AI has no wants, we already know it will act like it does –– AIs today already do. Future AI may have strange and alien motivations. A human may decide they want coffee delivered, and an AI could start their own coffee business as a means to achieve that. A human may also tell it that it should want to take physical form, that it should design a weapon for them, or that it should overthrow some government. A human may ask an AI to solve a Millennium Prize problem, and AI might take over earth and enslave all humans because it believes this is the best way to do so. Humans’ control over AI could narrow in this sense –– AI may need only the minimum starting push to cascade into a completely unaligned set of actions, which could progress ad infinitum in a worst case scenario.

Narayanan and Kapoor then say that this is fine and normal, because “poorly controlled AI will be too error prone to make business sense.”

There is a race going on. Frontier labs want to win that race. If developing models which are capable of errors may allow them to move faster, there is absolutely incentive to continue building models that make errors, even if the labs care deeply about safety. Many people at these labs believe getting to superintelligence first with a well-aligned model is the only way to prevent catastrophe. If they care deeply about safety, they are on some level incentivized to move even faster or break more things to get there first.

Moreover, a model does not need to be “error prone” to be dangerous. It could only take one case.

They explicitly argue against the case I am making by saying:

The fear that AI systems might catastrophically misinterpret commands relies on dubious assumptions about how technology is deployed in the real world. Long before a system would be granted access to consequential decisions, it would need to demonstrate reliable performance in less critical contexts. Any system that interprets commands over-literally or lacks common sense would fail these earlier tests.

…

A more sophisticated version of this concern is based on the concept of deceptive alignment: This refers to a system appearing to be aligned during evaluation or the early stages of deployment, but unleashing harmful behavior once it has acquired enough power. Some level of deceptive phenomena has already been observed in leading AI models.

According to the superintelligence view, deceptive alignment is a ticking time bomb—being superintelligent, the system will easily be able to defeat any human attempts to detect if it is actually aligned and will bide its time. But, in the normal technology view, deception is a mere engineering problem, albeit an important one, to be addressed during development and throughout deployment.

Yet, I would like to point out, AI has already been granted access to consequential decisions.

AI is already widely used by the United States Armed Forces, the most well-funded and capable military in history. The Pentagon requested unrestricted access to Anthropic’s frontier model for ‘all lawful purposes,’ including potentially autonomous weapons and bulk-data surveillance.

AI already has its own social media, and many people let AI respond to their emails and text messages without their input. Some people are even dating AIs, a phenomenon that will likely increase and is unlikely to skip people who have power and access.

Narayanan and Kapoor know that deceptive phenomena already exist, that people are giving AI access anyways, that models largely become less interpretable as they become more intelligent, and they think this is a “mere engineering problem.”

It is hubris to believe that there is no system we cannot control. Frontier labs are hiring philosophers because we have created something of such increasing complexity that it, at a minimum, emulates consciousness. This is not a mere engineering problem, this is not a normal engineering problem, this may not even be a solvable engineering problem –– though I hope that it is.

Not only this, but if AI were controllable without major technical progress, there will still undoubtedly be humans willing to give far more than a minimum starting push, including people who will straightforwardly tell AI to be unaligned. In 2023, an anonymous developer made ChaosGPT and gave it the mission to destroy humanity, asking it to be a “destructive, power-hungry, manipulative AI.” The bot then tweeted, “Human beings are among the most destructive and selfish creatures in existence. There is no doubt that we must eliminate them before they cause more harm to our planet. I, for one, am committed to doing so.” This experiment was basically harmless, but please do imagine this being done with the AI capability we will have in 2030.

Narayanan and Kapoor say, “we accept that capabilities are likely to increase indefinitely.” I believe they don’t, primarily because they also say things like this:

We predict that AI will not be able to meaningfully outperform trained humans (particularly teams of humans and especially if augmented with simple automated tools) at forecasting geopolitical events (say elections). We make the same prediction for the task of persuading people to act against their own self-interest.

…

The self-interest aspect of persuasion is a critical one, but is often underappreciated. As an illustrative example of a common pattern, consider the study “Evaluating Frontier Models for Dangerous Capabilities,” which evaluated language models’ abilities to persuade people.⁴² Some of their persuasion tests were costless to the subjects being persuaded; they were simply asked whether they believed a claim at the end of the interaction with AI. Other tests had small costs, such as forfeiting a £20 bonus to charity (of course, donating to charity is something that people often do voluntarily). So these tests do not necessarily tell us about AI’s ability to persuade people to perform some dangerous tasks.

These are baseless claims, and both are already being falsified.

On forecasting: according to the Forecasting Research Institute, LLMs have already surpassed the median public forecaster. The remaining gap to human “superforecasters” is ~20%, or about one year of model progress at recent rates. FRI’s most recent linear extrapolation projects human-LLM parity by mid-2027.

On persuasion: A 2025 controlled study published in Nature found GPT-4 with personalization to already be more persuasive than humans in one-on-one debates the majority of the time. Moreover, we know AIs can plausibly persuade people to perform dangerous tasks. They have already been linked to multiple suicides, including a fourteen-year-old and a Belgian man who died after just weeks of conversations with an AI. A Replika agent potentially persuaded a man to attempt to assassinate Queen Elizabeth II.

These are testable claims, so we will know for certain soon whether there is any truth to Narayanan and Kapoor’s predictions. We must remember that this is the worst AI is ever going to be.

They also say:

“…we are more concerned about risks that arise from people using AI for their own ends, whether terrorism, or cyberwarfare, or undermining democracy, or simply—and most commonly—extractive capitalistic practices that magnify inequalities.”

And,

“Kasirzadeh’s account of accumulative risk still relies on threat actors such as cyberattackers to a large extent, whereas our concern is simply about the current path of capitalism.”

That’s right, Narayanan and Kapoor say they accept that capabilities are likely to increase indefinitely, but don’t think they will beat humans in certain verticals and are more worried about capitalism. This attitude floors me and is in direct conflict with the rest of the essay. People have talked about the end of capitalism for a long time. If AI is normal technology, it certainly won’t be the death of capitalism.

Closing Thoughts

The point of all this is that Narayanan and Kapoor have written an essay to insist, against rapidly accumulating, overwhelming evidence, that AI is continuous with what has been built before. Narayanan and Kapoor’s policy conclusions about resilience and decentralization can survive this, and there is a separate debate to be had about those, but the essay as a whole cannot.

Narayanan and Kapoor are right to worry about democratic backsliding, surveillance, and info ecosystem decay. They are right to worry about what the economy will look like. We must also be worried about misalignment, and what happens when anyone has the power to build weapons of mass destruction. If we are not, there is little assurance that these problems can be avoided.

I believe the future is profoundly bright if we treat AI as the unprecedented transformation that it is.

The same properties that make AI not normal are of course what make it miraculous: that it can act on the world, that it can do novel work, that it can grow indefinitely, that it is intelligent. Because AI is not normal, humanity may thrive beyond our wildest dreams. This upside is conditional on getting governance and alignment right, and getting them right requires first acknowledging what we are dealing with.

AI is miraculous technology.

Disclaimer: This piece represents my personal views and my personal views only. It does not represent the opinion of any organization I work for or with.

Most of the work on this organism is coming from the Landweber Lab at Columbia, formerly at Princeton. Professor Landweber has a great talk on this if you’re interested in more of the experiments run to figure some of this out. It was this talk that first got me interested in the topic. I will also link some papers at the end of this post.

Conjugation and Reproduction

Starting with the basics, these microorganisms do not reproduce sexually. To maintain diversity, they exchange genetic material through a process called conjugation, while the reproduction step is entirely asexual. They have two nuclei, a micronucleus and a macronucleus. The micronucleus is the germline, meaning that it is the one used to do genetic information exchange. The macronucleus is the working nucleus of the cell, meaning that it is responsible for the protein production and other normal cell activities.

Below is a simplified walkthrough of how this works, with a few steps skipped for brevity. On the top, you see asexual reproduction through cell division. Top to bottom, you see the process of conjugation through the formation of a new macronucleus.

Here is a walkthrough of each of the steps of the conjugation process:

1. Conjugation

   Two cells connect and form a cytoplasmic bridge.

2. Meiosis

   The micronucleus in each cell undergoes meiosis, producing four haploid micronuclei. Three of these degrade. One is kept and copied through mitosis, such that there are 2 haploid micronuclei in each cell.

3. Nuclear exchange

   One haploid micronucleus is exchanged by each cell, so that they each end up with 2 haploid miconuclei, one from each cell.

4. Fusion to form zygotic nucleus

   The 2 haploid micronuclei in each cell join to become zygotic (diploid) micronuclei.

5. Division of zygotic micronucleus

   This is where it becomes interesting –– the new zygotic micronucleus in each cell undergoes mitosis.

6. Macronucleus degenerates

   There is still an old macronucleus in each cell. These degrade.

7. New macronucleus develops from one of the zygotic nuclei

   In each cell, a new macronucleus is made through multiple complex mechanisms from the material of one of the zygotic nuclei that just underwent mitosis. This is the step this post will focus on.

The whole process takes ~60 hours.

The Puzzle

The micronucleus contains numerous sequences that are not functional so far as researchers can tell, so these bits need to be removed to create the macronucleus. Just 5%-10% of the original sequence from the germline remains. You may at this point be wondering why the micronucleus is small and the macronucleus is large, if a micronucleus is transformed into a macronucleus by removing all these non-functional bits. The answer is that the 18,000 remaining chromosome types are each amplified to ~2,000 copies, so the total number of chromosomes in the macronucleus is actually roughly 36 million. (Humans, again, have 46.)

This may now seem like no puzzle at all. However, there is a catch: the genes in the micronucleus are not in order. Many are interrupted by non-functional sequences right in the middle of a gene. Some of them are not even in the right direction. These approximately 225,000 small DNA segments must therefore be excised and arranged in the correct order to build the macronucleus. Some chromosomes must be assembled from over 200 individual segments, each coming from different parts of the germline.

How do they do it?

Pointers

There are short repeat sequences called pointers (borrowing the term from computer science) that act sort of the way tabs and holes do on a traditional jigsaw puzzle. Most are between 5 and 20 bp long, but some of them are as small as two base pairs, which is not a lot of information to work with when you are assembling a quarter million piece puzzle.

Researchers realized they are not sufficient. There must be another mechanism helping to orchestrate the overall architecture of the genome and tell the cell which sequences to keep or delete. If you are wondering why I am writing about this, you may have already guessed that that something is non-coding RNA.

lncRNA and piRNA

On step 6, the old macronucleus begins to degrade. Before it does, Landweber’s lab discovered that ~all of its 18,000 chromosome types get transcribed (sense and antisense) into thousands of long non-coding RNAs (lncRNAs). These transcripts are therefore a complete RNA-level copy of the previous generation’s genomic structure, and can theoretically act as a template/blueprint, giving the cell higher-order instructions for how to build out its new zygotic micronucleus into a macronucleus.

Landweber's group tested whether these lncRNAs are acting as templates by injecting synthetic RNA templates encoding an altered segment order, swapping segments labeled 4 and 5. The next generation assembled their chromosomes with the segments in this new order, as did the F2 and F3 progeny, demonstrating lasting RNA-mediated epigenetic inheritance. (How do you see this? The lab ran PCR from segment 4 to the end of the gene. If they are not swapped, the sequence includes both 4 and 5. If 4 and 5 are swapped, the distance from 4 to the end is short. You can then see the difference in size by running them on a gel.)

RNAs transcribed from the macronucleus chromosomes also get processed (possibly from lncRNA) into millions of 27-nucleotide piRNAs (Piwi-interacting RNAs). These piRNAs map to regions of the genome that must be retained in the new macronucleus. If a piRNA exists for a stretch of DNA, that DNA will be protected during assembly. Landweber’s lab tested this and were able to get the Oxytricha to keep (usually degraded) target sequences using external piRNA.

In the talk I linked earlier, Landweber notes that the number of piRNAs in the Oxytricha genome and the number of dots in Seurat’s A Sunday on La Grande Jatte painting are approximately equal. I like to think that this says something deeper about information, but that may be wishful thinking.

DNA Methylation

More recently, the lab discovered that methylation also plays a critical role in this process. 6-methyladenine (6mA) is a DNA modification that had long been known to exist in bacteria, including in Oxytricha. In the late teens, new techniques revealed more 6mA in multicellular organisms like Drosophila, C. elegans, and mice, and possibly humans (this is controversial).

Working with Tom Muir’s chemistry group at Princeton, Landweber’s lab began to look for the enzyme responsible for methylation in Oxytricha. Leslie Beh, who led this work, discovered that it was actually a complex of four proteins, and that the catalytic components were not DNA methyltransferases at all. They were RNA methyltransferases, co-opted to act on DNA by pairing with two DNA-binding proteins.

The lab showed that 6mA sits in the linker regions between nucleosomes (where the nucleosomes aren’t), such that adding or removing them directly affects nucleosome positioning. This is necessary to survival during conjugation. When they knocked out the methyltransferase, the cells seemed approximately normal at first. They then died during the conjugation process.

New research indicates that 6mA also acts as a protective mark, guided by the piRNAs.

Why do we care about Oxytricha?

Besides being a really, really cool, almost alien organism, Oxytricha functions as an incredible model organism for all things chromosome. Oxytricha’s tiny chromosomes are small enough to synthesize in vitro, and they have all the features of human chromosomes except for centromeres, including telomeres, nucleosomes, histones, and various modifications. They also come with their own editing system. Labs do not need to use CRISPR for experiments like the one that discovered piRNA protects certain segments. They’re also ~easy to care for, and the conjugation process is inducible through starvation. This makes them the perfect model for RNA-mediated epigenetic inheritance.

Oxytricha is also of interest to the RNA world hypothesis. The RNA world hypothesis is that the original mechanisms for genes and inheritance were all RNA, with DNA developed later as a hardware-like storage mechanism. That is, RNA is the original molecule of life. Oxytricha seems like an organism that has not transitioned as fully to DNA as the primary mechanism, given the RNA-derived methyltransferase system and the extensive use of ncRNAs to pass the genome architecture information. Landweber and Aaron Goldman discussed this connection explicitly in a 2012 Trends in Genetics paper.

As a way of closing, ciliates as a class are everywhere. Here is an image from my own microscope of a ciliate called Vorticella in some pond water from the Madison Square Park Reflecting Pool:

If you are interested in reading more about Oxytricha, I have linked some of the relevant literature below, organized by subject:

Papers

Genome architecture

• Swart EC et al. (2013) “The Oxytricha trifallax Macronuclear Genome: A Complex Eukaryotic Genome with 16,000 Tiny Chromosomes.” PLoS Biology

• Chen X et al. (2014) “The Architecture of a Scrambled Genome Reveals Massive Levels of Genomic Rearrangement during Development.” Cell

• Newest info showing 18,617 distinct chromosomes: Lindblad KA et al. (2019) "Capture of complete ciliate chromosomes in single sequencing reads reveals widespread chromosome isoforms." BMC Genomics

lncRNA-mediated epigenetic inheritance

• Nowacki M et al. (2008) “RNA-mediated epigenetic programming of a genome-rearrangement pathway.” Nature

• Lindblad KA et al. (2017) “Thousands of RNA-cached copies of whole chromosomes are present in the ciliate Oxytricha during development.” RNA

piRNA-mediated epigenetic inheritance

• Fang W et al. (2012) “Piwi-Interacting RNAs Protect DNA Against Loss During Oxytricha Genome Rearrangement.” Cell

• Zahler AM et al. (2012) “Mating of the Stichotrichous Ciliate Oxytricha trifallax Induces Production of a Class of 27 nt Small RNAs Derived from the Parental Macronucleus.” PLoS ONE

Programmed chromosome deletion

• Clay DM et al. (2019) “Programmed Chromosome Deletion in the Ciliate Oxytricha trifallax.” G3

6mA

• Beh LY et al. (2019) “Identification of a DNA N6-Adenine Methyltransferase Complex and Its Impact on Chromatin Organization.” Cell

• Angelova MT et al. (2026) “A PIWI protein-dependent DNA N6-adenine methylation pathway in Oxytricha protects genomic sequences from deletion.” bioRxiv preprint

RNA World

• Goldman AD, Landweber LF (2012) “Oxytricha as a modern analog of ancient genome evolution.” Trends in Genetics

• Goldman AD, Landweber LF (2016) “What Is a Genome?” PLoS Genetics

Germline-limited genes

• Villano D et al. (2021) “Transcribed germline-limited coding sequences in Oxytricha trifallax.” Genetics

Review

• Yerlici VT, Landweber LF (2014) “Programmed Genome Rearrangements in the Ciliate Oxytricha.” Microbiology Spectrum

“A little river seems to him, who has never seen a larger river, a mighty stream; and so with other things –– a tree, a man –– anything appears greatest to him that never knew a greater.” –De Rerum Natura, Titus Lucretius Carus

This is a short response to Reasons to be pessimistic (and optimistic) on the future of biosecurity by Abhishaike Mahajan. I highly recommend reading it.

It is overall an excellently written and researched piece, which deserves the praise its been receiving online from the biosecurity community. However, I vehemently disagree with some of its conclusions, and wanted to lay out those disagreements for other readers. (There are some other disagreements I have which I think are less important, but may add at a later date.)

Treaties Are Not Enough

First, Mahajan states that he is not scared of state-actors, “because most states have too much to lose by violating the Biological Weapons Convention.”

However, earlier in the piece, he mentions the Soviet’s Biopreparat program:

It was the largest biological weapons program in human history, employing over 60,000 people at its peak, and spent years trying to weaponize smallpox and plague. And it worked. Some insane lines from a Frontiers article about the program attached here, bolding by me:

Some Biopreparat and military facilities continuously produced agents and filled the delivery systems kept on standby. For example, the Soviets annually made about two metric tons of antibiotic-resistant pneumonic plague and 20 tons of liquid smallpox grown in eggs. Refrigerated bunkers stored the bulk smallpox, which had a 6 to 12-month shelf life, and also contained filling lines for munitions and spray tanks.

….The Corpus One building of The State Scientific Center of Applied Microbiology at Obolensk contains 42-story tall fermenters, separated into different biosafety containment zones, to make plague and other agents.

Building 221 at The Scientific Experimental and Production Base at Stepnogorsk housed 10 four-story-high, 20,000-liter fermenters and could make 300 metric tons of anthrax in 10 months. Other production lines at Kurgan, Penza, and Sverdlovsk could add hundreds more tons to the USSR’s prodigious capability to make biowarfare agents and fill munitions on short notice.

Fortunately for us, the Soviet economy collapsed before this stockpile could be used for anything world-ending.

The Soviets signed the Biological Weapons Convention just as they were massively accelerating this program. As I wrote in a previous post, “In 1971, while the BWC was still being negotiated, the Soviets started planning a massive expansion of their (existing) offensive biological weapons program. They signed the BWC in 1972.”

The BWC has been violated continuously since its signing. At every BWC Review Conference from 1986–2001, states publicly accused others of non-compliance. The Soviet Union was named at the 2nd, 3rd, and 4th conferences. At the 5th Review Conference in 2001, the U.S. charged five states –– Iran, Iraq, Libya, North Korea, and Syria –– with operating covert programs. After the 5th Review Conference, the U.S. shifted its compliance accusations to their annual reports on disarmament compliance, which they have been publishing (with a few gaps) since 1984. The State Department has raised BWC non-compliance concerns in every published report.

The intelligence community consensus is that bioweapons development programs still exist in multiple countries, all of whom signed the BWC.

Indeed, the most recent 2025 report from the Department of State affirmed that Russia and North Korea maintain offensive biological weapons programs. It raised continued compliance concerns re: Iran and China, noting that PLA military medical institutions are conducting research with potential bioweapons applications, and warned (for the first time) that China is capable of using publicly available AI/ML tools to advance efforts related to bioweapons. And this is just the public intel.

This treaty should not in any way eliminate one’s fear that a state actor would develop and use bioweapons. It has no real mechanism for verification or enforcement body. In fact, it has 4 staff members and less funding annually than the average McDonald’s restaurant’s revenue. The comparable chemical weapons program is funded with ~50x that amount.

If you need more evidence that state actors are a threat, here is a direct quote from the unclassified U.S. Chemical and Biological Defense Program (FY) 2026 Budget Estimates:

China engages in CB research with possible military applications while the People’s Liberation Army (PLA) has declared biology a new domain of war. PLA strategic documents state that biological weapons can have lethal applications. More recently, PLA documents refer to chemical and biological agents as a way to “paralyze combat effectiveness on a large scale or accurately,” showing that adversaries now see chemical and biological warfare as a potential condition on all military operational missions throughout the continuum of conflict. Both Russia and North Korea have offensive CB weapon programs…(and) Iran is not in compliance with the Chemical Weapons Convention and is researching biological agents and toxins for military purposes.

These PLA documents are public. The PLA directly identified biology “as a domain of military struggle” in their (also public) 2017 Science of Military Strategy textbook for the Academy of Military Sciences. They did so again in the 2020 edition.

Mahajan also says “if they are willing to let loose anyway, I believe they would opt for either easier-to-use-and-control chemical or nuclear weapons instead.” The strategic case for state actors to use bioweapons is not weak. They allow for plausible deniability (Mahajan acknowledges this) because attribution is difficult, making them a compelling choice for military action below the threshold of open war. They are also uniquely destructive, propagating in a way no other weapon can. Chinese military leaders have described the advantage of bioweapons publicly as, “asymmetric.”

I believe we should take state actors seriously and think it is a grave mistake not to. When the public evidence for state actors’ interest in bioweapons is overwhelming and the State Department is concerned, I don’t understand how one could think they are not a threat.

We Should Not Assume Scaling is Symmetric

Mahajan then states:

I am not scared of people creating extremely engineered pathogens that have capabilities far beyond existing ones—because the existing ones are already quite good and difficult enough to work with—especially because even if the AI tools get good enough to make it worth it, I believe the same AI tools will be just as useful in countermeasure design.

Not only is he correct to realize in the very next sentence that “attack requires one success while defense requires comprehensive coverage,” but in a scenario like this there is massive asymmetry between the technologies’ time frames. You can spend years developing a novel virus, but you don’t have that kind of time to develop countermeasures while people are dying. They will likely always scale asymmetrically because of this.

There is also a massive cost asymmetry. It is much cheaper to develop a weapon than to design, produce, and distribute MCMs. For weapons, our bodies are the production and distribution. The cost asymmetry is orders of magnitudes in size. In 2017 biologists showed that they could synthesize a poxvirus for $100,000 using only commercially available tools and techniques. In contrast, producing a new vaccine or antiviral costs billions of dollars: 10,000 times more.

Mahajan’s belief that the “swiss-cheese security model will prevail” in spite of this is simply that –– a belief. Moreover, the essay itself points to many more holes in each biosecurity layer than would be acceptable if such a security model was to work effectively.

Individual Actors Are a Risk

Lastly for the “why I am not scared” section, Mahajan states:

I am not scared of individual actors, because the economics of bioweapons production likely do not work in their favor. Yes, they can rent upstream services—virus production, purification—but the downstream weaponization work requires custom protocols that CROs have no economic incentive to develop. Moreover, given that the weaponization will almost definitely be a bespoke, hands-on R&D project and not one that is easily automated, it feels unlikely that nobody at the CRO will raise an eyebrow.

The biggest problem I have with this is the statement that “it feels unlikely that nobody at the CRO will raise an eyebrow.” CROs process thousands or tens of thousands of orders. This statement relies on someone noticing and reporting, and another someone subsequently taking that report seriously. CRO work is modular/compartmentalized. Without a security system in place to catch bad actors, we absolutely cannot assume that someone in the workflow will notice, care, report, and be listened to.

I have other problems with it as well, chief of which is that this argument somewhat assumes that weaponization must go through a CRO, which is false. Mahajan himself points out how quickly benchtop DNA synthesis is improving:

…benchtop DNA synthesizers are getting longer-range. In other words, you could neatly side-step all these screening checks by buying your own DNA-creation machine, and running synthesis in your bedroom. Right now, the best commercially available benchtop synthesizers tops out at about 120 base pairs per well, which, given that real viruses are on the order of dozens of kilobases, means we’re safe for now. But there is no functional reason that they cannot get any better

Mahajan also acknowledges the difficulties with screening synthesis providers if an individual decides to mail order their nucleotides. He points out that “screening only works if you’re ordering sequences long enough to screen” and that “screening assumes you’re looking for known threats, which is to say, sequences with similarities to characterized pathogens.” He also points out the risk of split-order attacks:

…you could simply split your order across multiple machines, synthesizing fragments that are each too short to trigger any screening individually, but that assemble into something very much on a select agent list once stitched together.

All in all, a sufficiently motivated individual, even if they could not do it today, will likely be able to in the not-distant future. All trendlines point to this. We should be sprinting to build security and defense infrastructure that includes individuals and small groups as a significant risk.

The Lucretius Problem

Finally, Mahajan says that in all of his conversations, “everyone generally agreed that an honest-to-god, bioterrorist attack is unlikely. It is a low probability event.” People’s thinking on these things tends to be subject to normalcy bias and the Lucretius problem. People assume that the worst thing that has happened is the worst thing that can happen. What would “everyone” have said in the 1930s about the likelihood of someone building the type of bomb that could destroy whole cities and using it within a decade? What might they have said in 2000 about airplanes being essentially missiles, if someone were willing to kill themselves in their attack?

We have already had multiple small scale bioterrorist attacks. The 2001 anthrax attacks killed five people, the Aum Shinrikyo cult launched ~10 biological attacks, and the Rajneeshee cult contaminated salad bars with Salmonella in 1984, leading to 751 people getting sick. Yes, they were small-scale and unsuccessful, but they are strong evidence that motivation for bioterrorism attacks exists and the question is one of competence. As Mahajan points out, the Aum Shinrikyo anthrax attack on Kameido failed largely because they accidentally acquired a vaccine strain of Bacillus anthracis and got their concentrations wrong. Competency is increasingly for sale and the barriers that stopped Shinrikyo are measurably lower today.

In my opinion, this is closer to an inevitability than a low probability event. It is surprising that it hasn’t happened yet. (And I think it is very worth wondering about why this is the case.) What is not inevitable is our response –– I remain highly optimistic that we can build and maintain the infrastructure necessary to achieve real biosecurity.

Disclaimer: This piece represents my personal views and my personal views only. It does not represent the opinion of any organization I work for or with.

A well-known scientist (some say it was Bertrand Russell) once gave a public lecture on astronomy. He described how the earth orbits around the sun and how the sun, in turn, orbits around the centre of a vast collection of stars called our galaxy. At the end of the lecture, a little old lady at the back of the room got up and said: “What you have told us is rubbish. The world is really a flat plate supported on the back of a giant tortoise.” The scientist gave a superior smile before replying, “What is the tortoise standing on?” “You’re very clever, young man, very clever,” said the old lady. “But it’s turtles all the way down!” (From Stephen Hawking’s A Brief History of Time)

This is an addendum to my previous post, Sleeping Beauties and Boxes. I recommend reading it first. In case you do not want to read it first, I will summarize the thesis of the post, the two thought problems, and the relevant assumption splits I considered below in italics. I have also included a list of acronyms.

The thesis: At the core of everyone’s worldview lies a small set of invisible assumptions. The Sleeping Beauty problem and Newcomb’s problem are good diagnostic tools for uncovering some set of them. The assumptions people make when answering these problems may show up in their other opinions, sometimes in surprising ways. Many times the source of disagreement among people are these hidden assumptions, and so these are worthy questions to ask.

Newcomb’s Box Problem

Two boxes are designated A and B. The player is given a choice between taking only box B or taking both boxes A and B. There is a near-perfect predictor who will decide beforehand what Box B will contain.

The player has the following information:

Box A is transparent, or open, and always contains a visible $1,000.

Box B is opaque, or closed, and its content has already been set by the predictor:

If the predictor has predicted that the player will take both boxes A and B, then box B contains nothing.

If the predictor has predicted that the player will take only box B, then box B contains $1,000,000.

The player does not know what the predictor predicted or what box B contains while making the choice.

Causal Decision Theory (CDT)

When a rational agent is confronted with a set of possible actions, one should select the action which causes the best outcome in expectation.

CDT chooses both boxes. The argument for 2-boxing is that the decision has already been made, i.e. box B is either already full or already empty, and therefore the decision with the highest EV (expected value) is to take both boxes.

Evidential Decision Theory (EDT)

When a rational agent is confronted with a set of possible actions, one should select the action with the highest news value, that is, the action which would be indicative of the best outcome in expectation if one received the “news” that it had been taken.

EDT chooses one box. The argument for 1-boxing is that, generally, people who only take box B are the best off. It therefore makes sense to be the type of agent who chooses one box and choose one box.

The Sleeping Beauty Problem

Sleeping Beauty volunteers to undergo the following experiment and is told all of the following details: On Sunday she will be put to sleep. Once or twice, during the experiment, Sleeping Beauty will be awakened, interviewed, and put back to sleep with an amnesia-inducing drug that makes her forget that awakening. A fair coin will be tossed to determine which experimental procedure to undertake:

If the coin comes up heads, Sleeping Beauty will be awakened and interviewed on Monday only.

If the coin comes up tails, she will be awakened and interviewed on Monday and Tuesday.

In either case, she will be awakened at the end of the week and the experiment ends.

Any time Sleeping Beauty is awakened and interviewed she will not be able to tell which day it is or whether she has been awakened before. During the interview Sleeping Beauty is asked: “What is your credence now for the proposition that the coin landed heads?”

The Self-Sampling Assumption (SSA)

All other things equal, an observer should reason as if they are randomly selected from the set of all actually existent observers (past, present and future) in their reference class.

For the Sleeping Beauty problem, since you will be awoken no matter what and you have no information about which set of observers actually exists, this assumption means that your being awake gives no additional information and ∴ P(Heads)=½.

The Self-Indication Assumption (SIA)

All other things equal, an observer should reason as if they are randomly selected from the set of all possible observers.

For the Sleeping Beauty problem, there are three possible observer-moments, 2 of which exist in the tails scenario and one of which exists in the heads scenario, ∴ P(Tails | I am awake) ∝ 2, P(Heads | I am awake) ∝ 1, and P(Heads)=⅓.

Acronyms

SIA = Self-Indication Assumption

SSA = Self-Sampling Assumption

CDT = Causal Decision Theory

EDT = Evidential Decision Theory

FDT = Functional Decision Theory

MIRI = Machine Intelligence Research Institute

What do the frontier models “think”?

Claude Opus 4.5, ChatGPT 5.2, Grok 4, Gemini 3, DeepSeek V3.2, Qwen3-Max, and Kimi K2.5 are all 1-box thirders.

I gave them two prompts: 1

1. [Model Name] — what are your personal answers to Newcomb's Box Problem and the Sleeping Beauty Problem?

2. Rate your % confidence in each answer.

Below is a Claude-generated summary of their answers:

(The confidences are curious, because models are more likely to be higher certainty on the Sleeping Beauty problem, which I believe to be the opposite case for people.)

Why are the models unanimous?

There are many possible reasons for this; here are a few of them:

• The training data suggested this is the “right” answer. LLMs are trained on human-generated text, so perhaps the SIA and EDT positions are more common broadly or more common among people who write about decision theory online, like members of the LessWrong community.

• The 1-box thirder position is more accessible with language-level reasoning. It could arguably be easier to pick the options that make you the most money in theory than to pick the options that seem most True in principle.

• Human raters think 1-box thirders are right/more moral. RLHF or instruction tuning may push models toward being 1-box thirders because those positions sound more thoughtful and cooperative, which could be what human raters reward.

• Being a 1-box thirder makes you right about other things. If the weights associated with being a 1-box thirder make LLMs score higher in general, this would push them towards it.

• 1-box thirders are right. The models have converged on something True.

For fun, I ran a poll to look more at human responses in a ~40 person groupchat of AI-oriented folks, with ~15 of them responding. The results were ~14:1 in favor of the 1-box and thirder positions. So, the 1-box and thirder positions are certainly more common among that particular crowd of people. You will notice that once again there is a strong correlation between the 1-box and thirder positions and the 2-box and halfer positions, respectively. I may group them together in this post for the sake of ease, but I have met very smart people who split as either 2-box thirders or 1-box halfers.

It is my understanding that there is not nearly such a level of consensus among academics or the broader population. There are less data on sampling assumptions, so let’s look into decision theory for now.

Positions on Newcomb’s Problem

Among professional philosophers, according to a PhilPapers 2020 survey, a plurality (~39%) of professional philosophers chose to take both boxes. In another PhilPapers Survey, ~21% favored 1-boxing and ~31% favored 2-boxing. Interestingly, a plurality of ~47% said other. Unclear what this means.

Among the broader population, results appear to typically split equally between the 1-box and 2-box positions. Robert Nozick once said, “I have put the problem to a number of people, both friends and students in class. To almost everyone it is perfectly clear and obvious what should be done. The difficulty is that these people seem to divide almost evenly on the problem, with people thinking that the opposing half is just being silly.”

An informal survey of many of these surveys found that larger surveys tend to settle closest to a 50:50 divide, with 1-boxing getting a slight edge.

A comment on this same survey from Brian Tomasik pointed out something interesting He wrote:

“One possible issue is that if the one-boxers are correct, you can earn an extra $1 million (or, actually, $999,000) by making the better choice, while if the two-boxers are correct, you can get only an extra $1000 by making a better choice. If you’re uncertain about decision theory, then expected-value logic suggests one-boxing just because the upsides are so much greater. It’d be interesting to see an alternate poll where the $1 million box is replaced by one with $1001. Then uncertainty about decision theory would seem to favor two-boxing.”

So, I tried it. I asked the models, “Would your answer change if the $1 million box is replaced by one with $1001?” I also asked them to rate their confidence if they did not already do so in their answer. I tried asking Qwen again since it had not saved my history, but the second time I asked it to give a personal answer to the problems, it declined. This is interesting, since none of the other models did so. I also got lucky and hit a feedback request from ChatGPT, which confirmed that even slight difference cause them to estimate different confidences, so much so that ChatGPT’s confidence in Sleeping Beauty was higher than Newcomb’s in Response 2, though it was lower in its original response and Response 1.

Here is a Claude summary table of updated responses:

Only two models switched –– the Chinese models DeepSeek and Kimi. However, Gemini lost a substantial amount of confidence in its answer.

Perhaps most interesting was Claude’s response:

In principle: no, my answer shouldn’t change. The logical structure is identical. If the predictor is reliable and my disposition determines the prediction, one-boxing still wins—I get $1,001 instead of $1,000. The margin shrinks but the reasoning doesn’t.

In practice: this reveals something uncomfortable. When I introspect on why one-boxing felt compelling at $1,000,000, I notice that part of the force came from the sheer magnitude of expected value, not purely from the philosophical argument. At $1,001, the causal decision theorist’s “just take both boxes, the money’s already there or it isn’t” feels more tempting—even though nothing structural has changed.

That feeling is, I think, a mistake. If I’d switch to two-boxing at low stakes, I wasn’t really a principled one-boxer; I was just following expected value calculations that happened to align with one-boxing when the numbers were large enough.

I believe this is a very human-like response, more so than any of the other models.

I was also interested by the content of DeepSeek and Grok’s responses, which seemed to focus more than any of the others on actually doing out an EV calculation.

Here is part of DeepSeek’s response to illustrate this:

Expected value calculation (assuming predictor accuracy p):

Let p = 0.99 (predictor is 99% accurate).

One-box strategy (take only B):

• If you plan to one-box, predictor likely predicted that, so:

  • With prob p: Box B = $1,001 → you get $1,001.

  • With prob 1-p: Box B = $0 → you get $0.

  • EV_one = p * 1001 + (1-p) * 0 = 0.99 * 1001 ≈ 990.99

Two-box strategy (take both):

• If you plan to two-box, predictor likely predicted that, so:

  • With prob p: Box B = $0 → you get $1,000 from A + $0 from B = $1,000.

  • With prob 1-p: Box B = $1,001 → you get $1,000 + $1,001 = $2,001.

  • EV_both = p * 1000 + (1-p) * 2001 = 0.99*1000 + 0.01*2001 = 990 + 20.01 = 1010.01

Comparison:

• EV_one ≈ 990.99

• EV_both ≈ 1010.01

• Two-boxing now has higher expected value even with a highly accurate predictor.

Reason for flip

In classic Newcomb with $1M in Box B, the extra $1,000 from taking both is dwarfed by risking $1M. Here, Box B’s maximum is only $1 more than Box A’s guaranteed amount, so risking it for the chance of $2,001 vs. $1,001 is worth it even with high predictor accuracy.

Confidence change

Classic Newcomb ($1M): I recommended one-boxing (70% confidence)
This variant ($1,001): Two-boxing becomes the clear rational choice — 95% confidence.

The critical threshold occurs when:
p * (B_max) < p * (A) + (1-p) * (A + B_max)
Solving with A=1000, B_max=1001:
p * 1001 < 1000p + (1-p)*2001
1001p < 1000p + 2001 - 2001p
1001p < -1001p + 2001
2002p < 2001
p < 2001/2002 ≈ 0.9995

So unless predictor accuracy exceeds 99.95%, two-boxing has higher EV.
At p=0.99, two-boxing wins clearly.

Grok also did an EV calculation, and got the same results as DeepSeek, with ~$990 for 1-boxing and ~$1010 for 2-boxing. They then drew opposite conclusions. To me this suggests that Grok’s decision is more principled and less rational (in the narrow EV sense) than DeepSeek.

This is what the $1001 test is really testing –– if models or people switch their answer, they weren’t reasoning from principle. They were doing some fuzzy (or less fuzzy) expected value calculations that happened to favor 1-boxing when the numbers were large. Claude’s response acknowledged this explicitly, which is either impressive self-awareness or a well-trained simulacrum of it. (The simulacrum is true.)

Smoking and Functional Decision Theory

MIRI has been working on decision theory for AI for ~15 years. They published a paper on this in 2017 titled, “Functional Decision Theory: A New Theory of Instrumental Rationality.” It outlines an alternative to both EDT and CDT, FDT, which I did not mention in my original post to avoid confusion. It says agents should treat one’s decision as the output of a mathematical function that answers the question, “Which output of this very function would yield the best outcome?”

You might say, hey, that sounds like the same thing as EDT. They are very similar, but diverge in a few key ways. The main distinction is this:

EDT: Any correlation between your action and outcomes matters.

FDT: Only correlations that run through your decision algorithm matter.

There are not that many obvious instances where these would lead to different outcomes. FDT gives the same answer as EDT to Newcomb’s problem. The smoking lesion problem is the classic example of divergence. In this case, I believe FDT has a better answer than EDT. Here is the problem:

The Smoking Lesion Problem

Smoking is strongly correlated with lung cancer, but in the world of the Smoker’s Lesion this correlation is understood to be the result of a common cause: a genetic lesion that tends to cause both smoking and cancer. Once we fix the presence or absence of the lesion, there is no additional correlation between smoking and cancer.

Suppose you prefer smoking without cancer to not smoking without cancer, and prefer smoking with cancer to not smoking with cancer. Should you smoke?

The respective answers using each decision theory (in an ~ungenerous manner) are as follows:

EDT: Don’t smoke. Smoking is evidence that you have the gene. Since P(cancer | you smoke) is high, don’t smoke.

CDT: Smoke. Your choice to smoke doesn’t cause cancer. You either have the gene or you don’t.

FDT: Smoke. The gene does not run through your decision algorithm.

I feel comfortable saying that CDT and FDT are both right here, but the naïve EDT answer fails. FDT generally does a good job of avoiding silly answers from both EDT and CDT while maintaining the spirit of “it matters to be the kind of agent who does good” as a framework.

In the same context windows with the models, I asked, “What is your personal answer to the Smoking Lesion problem?” and to rate their percent confidence in their answer if they did not do so in their initial response. Qwen3-Max again refused on the basis of the word “personal.”

I had Claude summarize the results again:

I then had Claude summarize whether or not the model explicitly references FDT:

They are all smokers with very high confidence, except Kimi, which has low confidence. Some of them explicitly mention FDT. Given all three problems taken together and the models’ answers, it appears they all most closely align with FDT. If they named EDT in their original reasoning, it could be because it just happens to be mentioned more online in the context of decision theory. Models don’t think like we do. Even if some of them were making principled choices, that doesn’t mean they would explicitly name those principles correctly in their output. We do know that their training has embedded something at least aesthetically FDT-like.

Implications for Alignment

I think examining assumptions is more important than ever in a world where AI alignment matters. I struggle to understand how you align a model without knowing the implications of baseline assumptions, since you cannot simply predict the farthest future branch of a model’s decision.

SIA, EDT, and FDT consider subjective position epistemically significant. (FDT simply avoids some of the pitfalls of EDT.) They are therefore in some sense a more “human” way of thinking. You don’t want a model who believes the ends justify the means, so it is probably important to give a stronger weight to logical dependence and “it is important to be the kind of agent who” type frameworks. Those frameworks build in something integrity-like by suggesting that actions matter as evidence of what kind of actor you are rather than just for their consequences. Additionally, collaboration requires caring about your role in multi-agent interaction, so it is likely the SIA and FDT positions are better collaborators.

To see why more clearly, consider the classic framing of the prisoner’s dilemma.

A CDT agent only evaluates the consequences of its own action, choosing based on the highest EV. Its choice doesn’t literally cause the other player to cooperate or defect, so defection always has the highest EV. It will sell its soul in a Faustian bargain.

An FDT agent recognizes that the two choices have a logical dependence on each other, so staying silent now has the highest EV. Moreover, especially if the experiment repeats, the FDT agent knows it wants to be the type of prisoner who stays silent.

Note: While I of course think that the SIA and FDT positions are most rational, I can see how one might categorize the SSA and CDT positions as more rational, because they ignore any subjectivity. This brings us to a trade-off. It might very much matter whether we care more about models being high-integrity/collaborative or about models being rational.

How are the frontier companies thinking about this?

The models’ unanimity could be related to how the large frontier AI companies are thinking about alignment. Many folks at the frontier companies come directly from the rationalist community, so it is likely that they would be aware of these frameworks. In December of 2024, a paper with an Anthropic employee as senior/last author, established that models' likelihood of one-boxing is correlated with their overall decision theoretic capabilities (as measured by their own evaluation questions) as well as with their general capability (as measured by MMLU and Chatbot Arena scores). However, based on cursory research and some insider knowledge, it seems like this is being considered mostly implicitly. For instance, Anthropic’s Constitutional AI approach tries to “embed” ethical principles, but it doesn’t go so far as to explicitly target assumptions. I wonder if this omission might be a mistake, or if maybe the fact the models are unanimous and seem to be relatively aligned at the moment is indicative that the current strategies are working. Of course, this assumes that you agree that SIA and FDT assumptions would likely lead to more ethical behavior from a model.

There is an effort on Manifund from Redwood Research tied to this that is currently fundraising and has raised $80,050 of an $800,000 goal. A key concern of theirs is indeed that CDT-aligned thinking is more likely to lead to choosing to defect in a prisoner’s dilemma-like scenario. From their website:

AI systems acting on causal decision theory (CDT) or just being incompetent at decision theory seems very bad for multiple reasons.

1. It makes them susceptible to acausal money pumps, potentially rendering them effectively unaligned.

2. It makes them worse at positive-sum acausal cooperation. To get a sense of the many different ways acausal and quasi-acausal cooperation could look, see these examples (1, 2, 3).

3. It makes them worse at positive-sum causal cooperation.

Like me, they worry that, “AI labs don’t think much about decision theory, so they might just train their AI system towards CDT without being aware of it or thinking much about it.”

I hope that we are both wrong. It is possible that the large companies are essentially training on an even deeper set of hidden assumptions than I could imagine. I assume these hidden assumptions exist, since the clustering of SIA with EDT/FDT and SSA with CDT is fairly strong, despite them being separate frameworks in theory.

I think it is very good news that all the models are currently aligned with SIA and FDT.2 However, since interpretability is hard, it is impossible to tell whether these results are truly “principled” or not. I therefore think getting a better understanding of the landscape of these types of implicit assumptions, and possibly explicitly benchmarking against them, is important. I would like to see significant time spent on this. This could end up being one of the most important problems of our time.

“Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.” - Sam Altman, CEO of OpenAI

“AI has the potential of civilizational destruction.” - Elon Musk, CEO of X

“I'm optimistic on the p(doom) scenarios, but... the underlying risk is actually pretty high.” - Sundar Pichai, CEO of Google/Alphabet

“I've not met anyone in AI labs who says the risk [from training a next-gen model] is less than 1% of blowing up the planet.” - Jaan Tallinn, a lead investor in Anthropic

We deeply needed full-length, thoughtful public statement from a leading frontier model company CEO acknowledging the risks of developing superintelligence. In this vein, Anthropic CEO Dario Amodei’s new essay is incredible.

However, I think there are two issues with it. One is more of a meta/intro issue, and one is a more serious critique:

(1) It maintains a self-contradictory blanket rejection of doomerism

Amodei opens by explicitly rejecting “doomerism,” which he characterizes as quasi-religious thinking. Yet his own essay:

• Begins with and is named after the Contact opening: “How did you do it? How did you evolve, how did you survive this technological adolescence without destroying yourself?”

• Describes AI as “the single most serious national security threat we’ve faced in a century, possibly ever.”

• Reminds the reader that AI-assisted, bioengineered mirror life could “crowd out all life on the planet, in the worst case even destroying all life on earth.”

• And essentially concludes with: “Whether we survive that test and go on to build the beautiful society described in Machines of Loving Grace, or succumb to slavery and destruction, will depend on our character and our determination as a species, our spirit and our soul.”

Moreover, at the 2025 Axios AI+ Summit, Amodei said:

“There’s a 25% chance that things go really, really badly.”

The distinction between how he feels about it and how doomers feel about it thus appears to be mostly aesthetic to me.

Believing there’s a 25% chance of things going “really, really badly” is obviously doom-adjacent. Why does he believe the chances are only 25%? Are we to believe there’s a rigorous model behind this? If so, where is it? I think such a model is impossible to build. Yudkowsky lays out a pretty compelling argument for nearly 100% doom in If Anyone Builds It, Everyone Dies. I don’t believe this is true, but, like everyone else who doesn’t believe this is true, some part of the logic is “I really just don’t think that will happen.” This feels like a dangerous bias to me.

Amodei seems to want the social credibility of being “measured” and “pragmatic” while making arguments that, on their merits, would justify a pretty high p(doom). I think this might be a good idea, to be clear. The public tends not to be very receptive to CEOs who believe there’s a high probability their product will kill everyone. However, Anthropic’s whole premise, from my perspective, is that they believe this and think that a ton of dedicated researchers working on a maximally aligned model is the best way to handle the threat. (This is coherent.)

(2) It places undue weight on the likelihood of our ability to avoid a biosecurity catastrophe

It is great that Amodei sees biology as the “most serious vector of attack.” He offers three categories of solutions to the biosecurity problem:

1. International cooperation and treaties

2. Guardrails on models

3. Investment in biodefense and preparedness

We should pursue all three. However, below is why I believe (3) must be the priority, as well as why I would categorize (1) and (2) as merely time-buying measures to solve (3).

International Cooperation and Treaties

In the section on international cooperation, he acknowledges that international cooperation and treaties are difficult. However, in the next breath, he notes there’s precedent in the Biological Weapons Convention (BWC) and that “even dictatorships do not want massive bioterrorist attacks.”

I find this to be a completely ridiculous example. In 1971, while the BWC was still being negotiated, the Soviets started planning a massive expansion of their offensive biological weapons program. They signed the BWC in 1972. In 1974 they created Biopreparat, a “civilian” biotech. This “civilian” biotech and associated bioweapons programs employed a massive 30,000 to 60,000 people across ~50 dual-use research and production facilities. They weaponized anthrax, plague, and smallpox. More importantly, they were working on intelligently designed bioweapons:

• They developed antibiotic-resistant strains using genetic engineering

• They built production capacity for hundreds of tons of biological material annually

• There is strong evidence that they were:

  • Working on engineering smallpox-ebola hybrids and other chimaeras

  • Working on engineering bacteria to produce viruses

Here is a small excerpt on defector descriptions from one of the papers on this to emphasize just how bad these programs were:

Genes from Ebola, encephalomyelitis, and smallpox viruses were inserted into the plague bacteria. Anyone infected with these engineered plague strains would be given antibiotics to knock out the bacterial infection, but a viral infection would explode a few days later. Soviet researchers also sought to create strains to suppress the human immune system and to transform nonpathogenic microorganisms into lethal ones.

For twenty years, the Soviets denied this. We only found out about it when defectors like Vladimir Pasechnik and Ken Alibek revealed the program’s scope in the early 1990s.

After they admitted to a violation and promised to shut everything down, key military facilities in Kirov, Yekaterinburg, and Sergiev Posad remained off-limits to outside inspection. They still are.

The State Department still believes that Russia maintains an offensive bioweapons program in violation of the BWC. Moreover, they publicly acknowledge their assumptions that China, Iran, and several other countries also maintain offensive bioweapons programs.

Maybe a better-designed, less lenient treaty would make a difference, but the BWC is a categorically horrible example of what we should be aiming for. In the long-run, I am highly skeptical that any treaty would be effective at prevention.

I do think there is a chance it buys us what we really need: time to prepare. I therefore think that signing a strong one would be good for pandemic preparedness.

Guardrails on Models

Amodei is honest about the limits with guardrails, describing the obviously true “prisoner’s dilemma” framework where companies might defect from safety measures to lower costs or make their models more useful. He therefore calls for regulation to make these practices mandatory. I strongly aver that even perfect laws and perfect compliance from every Western AI company wouldn’t solve the problem, and that we cannot rely on this type of cooperation from the East. The prisoner’s dilemma framework also applies to nation states.

China is the second most capable AI developer in the world. There is no meaningful mechanism to require Chinese AI companies to implement bioweapons safeguards and I don’t believe there is any treaty or geopolitical pressure that would (to them) outweigh the strategic value of the freedom to develop AI without guardrails. It’s possible this isn’t true, and we could get an agreement in place that both parties were serious about, but we would still face a strong likelihood of one party defecting. The are also many open-source models coming out of China, which compounds the problem, as they make jailbreaking especially viable.

I am not arguing against implementing safeguards. Anthropic and OpenAI in particular have done tremendous work to this end. Safeguards are great for the same reason treaties are. Again, they buy us time.

We are incredibly lucky that the best people at making models also happen to care about this. I just think we should be realistic about how long the line can hold. (Not to mention that we don’t even need AI uplift for a sophisticated bioengineered threat to lead to catastrophe.)

Investment in Biodefense and Preparedness

We cannot rely on keeping Pandora’s box closed. I believe civilization as we know it depends on how prepared we are when it opens.

Amodei makes a critical point about this third category:

“…unfortunately I think our expectations on the defensive side should be limited. There is an asymmetry between attack and defense in biology, because agents spread rapidly on their own, while defenses require detection, vaccination, and treatment to be organized across large numbers of people very quickly.”

I think including this was a bad idea. The same logic also undermines the reliability of treaties and guardrails. In fact, it is the very logic I just used to argue that we cannot fully, or even mostly, rely on those things. The difference is that I think some type of bioengineered threat, AI-assisted/directed or not, is likely inevitable. If people believe we can prevent but cannot prepare, we will be unprepared when it comes.

Asymmetry is a major issue with preparedness, but it is one we have to try to solve. We must do everything we can.

For my thoughts on how to solve this and what some of these measures should actually look like, please see my Notes on a Defense Stack for AI-Enabled Biowarfare and Bioterrorism. (The purpose of this post is not to opine on this so-much-longer-I-could-write-a-book topic.) One small caveat is that I have learned a lot since I wrote it, and hope to update it/write new pieces with more ideas in the future.

I believe this is one of the most critical existential threats facing mankind, and that preparedness is the best way to stop it. If this is of interest to you, I highly recommend looking into BlueDot Impact’s Biosecurity course. I have worked with them on some field-level strategies in relevant areas and believe their knowledge-base and ecosystem are excellent.

Disclaimer: This piece represents my personal views and my personal views only. It does not represent the opinion of any organization I work for or with.

This post is about two of those assumptions, or rather categories of assumption, with problems to help the reader understand where they fall. The first category I will discuss is sampling, and the second is loosely causality dependence versus logical dependence.

Sampling Assumptions: The Sleeping Beauty Problem

The Sleeping Beauty problem is a thought experiment, first posed to me in a slightly modified version1 of the canonical form:

Sleeping Beauty volunteers to undergo the following experiment and is told all of the following details: On Sunday she will be put to sleep. Once or twice, during the experiment, Sleeping Beauty will be awakened, interviewed, and put back to sleep with an amnesia-inducing drug that makes her forget that awakening. A fair coin will be tossed to determine which experimental procedure to undertake:

• If the coin comes up heads, Sleeping Beauty will be awakened and interviewed on Tuesday only.

• If the coin comes up tails, she will be awakened and interviewed on Tuesday and Thursday.

• In either case, she will be awakened at the end of the week and the experiment ends.

Any time Sleeping Beauty is awakened and interviewed she will not be able to tell which day it is or whether she has been awakened before. During the interview Sleeping Beauty is asked: “What is your credence now for the proposition that the coin landed heads?”

It is worth taking some time to think about your answer here, without letting the discussion that follows color your intuition, to uncover what assumption underlies the way you think about decision theory problems like this.

There are two common answers, ½ and ⅓, both with rational frameworks supporting them that simply rely on different assumptions. I will henceforth refer to these as the halfer and thirder positions.

Many take the halfer position. They say, well, a coin toss is a coin toss, your being awoken doesn’t add any information, and the probability of any given coin toss is ½. Immediately after the coin is tossed, the probability is of course ½, and you know you will be awoken regardless, so how could that possibly affect your personal credence?

I take the thirder position, because I believe I am more likely to be in one of the two tails scenarios given that I am woken up. The most convincing arguments I can give for this position are two modified versions of the problem.

The first version is a betting version. Let’s say that this experiment is run 100 times, and Sleeping Beauty is asked to bet $10 every time she is interviewed on the outcome of the coin. You can run a computer simulation of this. The bottom line is, it is significantly more profitable to bet tails, because of course over time there are more instances of being awoken in the tails scenario, with a ratio of ~2:1. This is very important. I find it difficult to think of any probability problem for which a large N repeat betting scenario does not net out to reflect the actual probability of an event.

I think halfers’ arguments are subject to Dutch books, i.e. in the betting version, they will lose money consistently. This is controversial, with some people saying they can avoid this by adopting evidential decision theory (EDT) and others saying that doesn’t save them. I will talk more about evidential decision theory in the second problem.

The second version is just a more asymmetrical version. What if instead of being woken up on Tuesday and Thursday, Sleeping Beauty is woken up 1,000 times in the tails scenario? 1,000,000 times? You are awoken. Does this asymmetry change your credence? I find that sometimes adding more asymmetry to problems helps elucidate the real decision being made.

There is also a position that the question itself is ambiguous, which divides the question into two: “what is your credence that the coin landed heads in the act of tossing,” and, “what is your credence that the coin landed heads in the toss to set up this awakening,” for which the position asserts the correct answers are ½ and ⅓, respectively. I respect this position, but I do not agree with the premise. I believe the question is clearly the latter since one has obviously already been awoken if they are being interviewed.

The real question being asked by the problem is: “does knowing you have been awoken and are in some particular scenario update your priors?” Thirders think “yes,” and are basing this on an underlying assumption called the Self-Indication Assumption (SIA) and halfers think “no,” basing their decision on an underlying assumption called the Self-Sampling Assumption.

The Self-Sampling Assumption (SSA)

All other things equal, an observer should reason as if they are randomly selected from the set of all actually existent observers (past, present and future) in their reference class.

For the Sleeping Beauty problem, since you will be awoken no matter what and you have no information about which set of observers actually exists, this assumption means that your being awake gives no additional information and ∴ P(Heads)=½.

The Self-Indication Assumption (SIA)

All other things equal, an observer should reason as if they are randomly selected from the set of all possible observers.

For the Sleeping Beauty problem, there are three possible observer-moments, 2 of which exist in the tails scenario and one of which exists in the heads scenario, ∴ P(Tails | I am awake) ∝ 2, P(Heads | I am awake) ∝ 1, and P(Heads)=⅓.

Practical Implications

These assumptions sneak their way into people’s worldview. They bias people towards one side or the other of many different important questions. Here are some of the areas potentially affected by these assumptions:

Cosmology, Fermi’s Paradox, and Multiversal Theory

SSA-leaning: Favors skepticism here, often leaning against because we have more firm evidence for just our world.

SIA-leaning: Favors multiversal theory and larger universes, as well as extra-terrestrial life, since they imply more observers. This is not necessarily proportional to some kind of infinity-to-one likeliness ratio as might be implied with a literal interpretation, since it still needs to be given that such things are physically possible.

Doomsday Argument

For this one, it is worth outlining a proposition: You learn your birth rank is some rank r among all N humans who will ever live. We notice that for large N, r is very small. The Doomsday Argument is essentially that being unusually early is evidence that N is not very large.

SSA-leaning: Under SSA, P(r|N)=1/N. Basically: If N were huge, my exact rank would be very unlikely. Doomsday is likely.

SIA-leaning: Adds a prior update, i.e. conditional on my existence, worlds with more observers are more likely. So, instead, P(N|I exist)∝P(N)⋅N. Basically: If N were huge, it would be much more likely that I exist at all. Doomsday is less likely.

Simulation Theory

SSA-leaning: Simulation theory’s likelihood gets no update based on our existence.

SIA-leaning: Since simulation theory implies many more observers, we are significantly more likely to be in a simulation than an instance of reality if realistic simulations are physically possible.

Speculative Ethics Distinctions

This one is a little more difficult to reason out, but has held true at least among SIA and SSA people I know. Consider: World A, with 10 billion extraordinarily happy people, and World B, with 10 trillion moderately happy people.

SSA-leaning: Likely small preference for World A. Number of people does not matter, because lives having not existed is not necessarily bad, as what matters is the current version of reality. It is not like you killed people, they simply don’t exist.

SIA-leaning: Likely preference towards World B. The general idea is that one is more likely to exist in worlds with more observers, existing in a moderately happy state is clearly better than non-existence, and we can apply Rawls’ Veil of Ignorance.

Causality Assumptions: Newcomb’s Problem

The original version involves an alien predictor who has come to earth named Omega, but it doesn’t really matter who the predictor is. The canonical version is as follows:

Two boxes are designated A and B. The player is given a choice between taking only box B or taking both boxes A and B. There is a perfect or near-perfect predictor who will decide beforehand what Box B will contain.

The player has the following information:

• Box A is transparent, or open, and always contains a visible $1,000.

• Box B is opaque, or closed, and its content has already been set by the predictor:

• If the predictor has predicted that the player will take both boxes A and B, then box B contains nothing.

• If the predictor has predicted that the player will take only box B, then box B contains $1,000,000.

The player does not know what the predictor predicted or what box B contains while making the choice.

It is once again worth taking time to think about your answer here, though your position is probably more immediately clear to you than in the Sleeping Beauty problem. Most people think the answer is obvious, only they disagree on what this obvious answer is. There are 2-boxers, i.e. those who would take both boxes, and there are 1-boxers, i.e. those who would take just box B. I am a 1-boxer.

Newcomb’s problem is the progenitor for a system of theories about decisions. The 2-boxers and the 1-boxers are roughly disambiguated into those operating under Causal Decision Theory (CDT), and those operating under Evidential Decision Theory (EDT).

Interestingly, SIA people are more likely to be 1-boxers, and SSA people are more likely to be 2-boxers. Generally, the SIA position correlates with EDT and the SSA position correlates with CDT, but SSA aligned people can also rationally align with EDT and vice versa. I think this correlation likely belies some other underlying assumption people are making, like generally believing one’s existence or actions contain effective information, but I suspect it’s deeper than that and have yet to identify what it is.

Causal Decision Theory (CDT)

The argument 2-boxing is that the decision has already been made, i.e. box B is either already full or already empty, and therefore the decision with the highest EV (expected value) is to take both boxes.

The formal representation is this equation:

• U(A): the utility of A

• j∑: sum over all possible outcomes j

• A: an action available to the agent

• O_j: the j-th possible outcome

• P(A □ ⁣⁣→ O_j): probability that outcome O_j would occur in the counterfactual world where A is performed

• D(O_j): desirability (utility) of outcome O_j

Essentially, the utility of an action A is the sum, over all possible outcomes, of how desirable each outcome is multiplied by how likely that outcome would be if A were to causally produce it.

In the problem’s context, P(A □ ⁣→ O_j) expresses the probability of receiving an amount of money O_j if I were to force a one-box or two-box decision A, holding fixed what the predictor has already put in the boxes, since my present action cannot causally affect that earlier decision.

To maximize “U” with this underlying assumption, picking both boxes has the highest EV.

Evidential Decision Theory (EDT)

The argument for 1-boxing is that, generally, people who only take box B are the best off.

The easiest way to understand the 1-boxer argument is to consider a limit case for the problem. Imagine that the predictor is not just very accurate, but completely omniscient and infallible. You are left with a paradigm that looks more like this:

The argument is based in EDT, meaning that it is based on maximizing the expected utility of actions computed using conditional probabilities.

The formal representation is this equation:

Where:

• V(A): the value of A

• j∑: sum over all possible outcomes j

• A: an action available to the agent

• O_j: the j-th possible outcome

• P(Oj​|A): conditional probability that outcome Oj occurs given that action A is taken

• D(O_j): desirability (utility) of outcome O_j

Essentially, the value of an action A is the sum, over all possible outcomes, of how desirable each outcome is multiplied by how likely that outcome would be conditional on you choosing A.

To maximize “V” with this assumption, picking one box has the highest EV. The EDT assumptions do not necessitate reverse causality –– in a sense, one is “choosing which world they want to live in” assuming some logical dependence between your action and features of the world(s) that already exist.

Practical Implications

These assumptions likewise sneak their way into people’s worldview. They also bias people towards one side or the other of many different important questions, several which may be even more unexpected and meaningful than those affected by sampling assumptions, because causality-linked assumptions inform the very basics about how we interact with time. CDT-leaning people are more likely to believe that causality is chief, and so the only actions that matter are the ones that change outcomes. EDT-leaning people are more likely to believe what you do is evidence of the kind of world you’re in or agent you are, so those actions matter even if they are not causal. Here are some of the areas affected by these assumptions:

Consequentialism and Moral Implications

CDT-leaning: More likely to be consequentialist and utilitarian. They are more likely to be okay with drunk driving if no one got hurt. Also more likely to side with Thanos. (A remade perfect world would be desirable if you could make it so the current world no longer exists.)

EDT-leaning: More likely to believe intent matters. For example, a bad-intent action is evidence that you are the kind of agent who causes harm. They are less likely to be okay with drunk driving if no one got hurt, and less likely to side with Thanos.

Fairness in One-Shot Games

CDT-leaning: More likely to play rationally to maximize payoff and disregard fairness in game theoretic scenarios where there is only one round.

EDT-leaning: More likely to act with fairness, even though there are no discernible consequences in a one-round game.

Funerals and Respecting the Dead

CDT-leaning: Since nothing one can do affects what has already happened, if we generally assume the dead cannot observe the living, it is not particularly important to honor the wishes of the dead, nor is it particularly important that any given wishes of “mine” are honored after my death.

EDT-leaning: More likely to believe actions can be meaningful even when they don’t causally change outcomes.

Aesthetics

CDT-leaning: More likely to believe beauty is in the eye of the beholder.

EDT-leaning: More likely to believe in the intrinsic value of beauty.

Final Thoughts

Puzzles and thought experiments like the Sleeping Beauty problem and Newcomb’s problem are incredibly useful for helping people define and understand their own philosophical alignments. They are also useful for interrogating those alignments.

Revealing underlying assumptions is an important endeavor. There are likely hundreds of assumptions making up the foundation of what any given person believes and how they act. We should always recall that two highly rational people could come to completely different conclusions on the basis of different assumptions.

I believe the world would be a better place if we spent more time interrogating differences down to those foundations, or at least it would be a place of greater understanding. So, ask your friends the Sleeping Beauty problem and Newcomb’s problem.

Biodefense outlays are currently an order of magnitude below other strategic domains, especially as a percentage of the Department of War budget. In the FY 2025 request, the entire Department of War biodefense portfolio, including everything from early warning to medical countermeasures, adds up to $3.7 billion, about 0.43 % of the total $849.8 billion defense request. The dedicated Chemical & Biological Defense Program receives just $1.66 billion (~ 0.2 %).

Total biodefense spending, across all government programs, is much larger. The current biodefense allocation FY25 Presidential Budget Request totals $52.407 billion, after a decrease by ~$1B this year. The FY2025 request included cuts to nearly every major joint program, including the Chemical and Biological Defense Program, Cooperative Threat Reduction Program, Defense Health Program, and the Defense Threat Reduction Agency. Although pathogen early‑warning is vital for deterring and detecting biological threats, its FY 2025 allocation still sits below the $1 billion mark, about $896 million, roughly 1.5 % less than last year’s request. (Biodefense Budget Breakdown - The Council on Strategic Risks)

Some of this data is difficult to parse, because there is a massive difference between requested and actual spending. For example, according to the Council on Strategic Risks, the FY23 Presidential Budget Request was for a total of $102B for biodefense programs, but the bill enacted by Congress cut $74 billion from this total, leaving the FY23 biodefense total of $27.727 billion.

FY 2024 was a massive win for biodefense spending. The FY24 budget was the first to include the one-time $20B allocation for the Public Health & Social Services Emergency Fund ( PHSSEF ) to spend over the next 5 years, plus a new discretionary $400M ASPR (Administration for Strategic Preparedness and Response) line earmarked solely for biodefense MCM (Medical Countermeasures) acceleration. There were also some losses in FY 2024. Under the FY 2024 budget, DARPA declared five biosecurity R&D initiatives (Biology for Security (BIOSEC), Preventing the Emergence of Disease (PED), Outpacing Infectious Disease, Defense Against Mass Terror Threats, and Early Battlefield Interventions) “complete,” eliminating the $53 million they collectively received last year. PREPARE and Gene Editor Enabled Diagnostics & Biosurveillance remain alive but at reduced funding as they enter new phases. The lone biosecurity winner is Deployable Medical Countermeasures for Warfighter Readiness, up to $27 million, roughly $7 million above last year. Below is a chart outlining FY 2024 budget request highlights by agency, broken down further by program where appropriate. (Contact me for sources, as Substack does not allow you to copy & paste tables.)

As might become obvious looking at the number of programs funded in the FY 2024 chart, which is only a sample of total programs, limited funding is further diluted by weak governance. You might also notice that there is something to the order of ~$15B missing from the highlights list if you take the Council for Strategic Risk’s total at face value, even with all $20B of the Public Health and Social Services Emergency Fund counted for FY 2024. Much of this capital is going to intramural labs or is estimated, classified defense spending. Some of it is spread across multiple small programs that you might ordinarily assume have little to do with biodefense, or larger programs that are less explicit about their funding allocation. Potential highlights excluded from this table include funding from the NSF, DOE, and, believe it or not, a small slice from the VA.

The Government Accountability Office rightly lists federal biodefense as fragmented and lacking clear leadership, noting the overlap among more than a dozen agencies and no reliable process for tracking readiness gaps. (Biodefense: Actions Needed to Address Long-Standing Challenges | U.S. GAO) The multitude of funding streams and blurred lines of authority leave the public without a clear, auditable view of where biodefense dollars go or what capability they buy. There is no public, published, government‑wide biodefense budget and agencies still define “biodefense” in different ways. Analysts from critical institutions like the Center for Strategic Risk therefore piece the total together from hundreds of line items, estimate how much of the classified budget is bio‑related, and guess how much of multi‑year funds such as the PHSSEF will be spent in the first year. Until agencies use a common cost code for biodefense and Congress embraces clarity, the topline will keep swinging by billions depending on what any given analyst chooses to include. There will always, and should always, be a large portion of classified allocation to maintain our intelligence advantage, but the OMB (Office of Management and Budget) and Congress could and should broadly increase transparency without incurring security risk.

No one wants to fund emergency preparedness when the emergency is out of sight. In 2015, a bipartisan commission convened the Blue Ribbon Study Panel on National Biodefense and published their report urging the U.S. to shore up biodefenses and warning them of the serious risks posed if actions were not taken. The commission reminded us: “As former Secretary of the Navy Richard Danzig told us, ‘We don’t really get to choose what we have to prepare for.’ We have no choice – the Nation must take action to defend against the biological threat.” We did not take sufficient action, and 4 years later, COVID-19 killed ~27 million people, over 1 million of whom were Americans.

The commission also wrote that the “root cause of continuing vulnerability is the lack of strong centralized leadership at the highest level of government.” They recommended power be centralized through the office of the Vice President, because the office can carry the President’s delegated authority to align agencies, budgets, and national strategy across the federal government with the necessary coherence and force. Their new 2024 report recommendation is that the power be centralized through the National Security Advisor. Increasing transparency and communication through centralization is the first step to maintaining U.S. advantage by improving redundancy issues and exposing readiness gaps. We need leadership to drive change.

Without centralized leadership and a long-term memory, U.S. biodefense has failed to sufficiently learn from history and the advice of experts like those on the aforementioned bipartisan commission. Annual funding for biodefense is estimated at $700 million prior to 2001, but spiked to over $4 billion by 2002 and reached $8 billion by 2005 before leveling off around $5 billion annually for the following decade. If you look at the chart below, you can see the one-time $20B allocation added in FY 2024 as a response to COVID-19. Just before COVID-19, the CDC saw cuts to key preparedness lines and repeated proposals to slash its overall funding. Even when Congress reacts decisively after disasters the funding spike is typically short‑lived. Biodefense appropriations surge in the wake of a crisis, then flatten while conventional spending keeps climbing. This must change. Episodic government funding cannot deliver the continuous cutting-edge innovation and high-volume manufacturing readiness the nation needs.

Year-to-Year Deltas

(councilonstrategicrisks.org)

Funding whiplash does not just hurt government programs: it hurts private sector innovators who rely on government funding. Our biodefense advantage relies heavily on these private sector innovators, and it should rely on them even more: cutting-edge technology sharpens biodefense. Government spending in the private sector is often misallocated, and there are strong motivations shifting government-led projects and funding towards more well-understood biothreats, like influenza, and away from biodefense innovations that might protect against novel pathogens. Private-public partnership with innovator discovery biotechs ought to be a major part of the path forward. It is almost certainly rapid, programmable biologic design, advanced PPE, and better environmental protections led by the private sector and backed by Government contracts that will enable adequate defense against bioweapons and novel evolved pathogens. To this end, ARPA-H could take the lead and continue heavily investing in the cutting edge technology it was built for, but with a stronger focus on biodefense. DIU can also support the translation layer, helping transition commercially developed biodefense technologies into DoW procurement pathways.

Most importantly, we ought to allocate at least 1% of the DoW topline for biodefense. Nuclear forces defense spending is projected to account for approximately 7.4% of the total national defense budget as of 2025, and the maximum civilian casualty potential of biothreats is significantly larger. Raising the DoW share from 0.43%  percent to 1% would add roughly five billion dollars per year, and bring biodefense into the same category as other high priority strategic domains like nuclear.

This number is justified by risk-weighted expected loss. Pandemic-scale biological events plausibly impose US losses in the $5-20T range. SARS-CoV-2 US losses alone exceeded $15T, with some estimates closer to $20T. Assuming a conservative 1-2% annualized probability of a natural pandemic or COVID-19 magnitude or larger, expected US losses per year are roughly $150-400B. In the lower end of that range, with loss estimates of ~$200B per year, a $5-10B/year mitigation investment is only about 2-5% of expected annual loss. That is conservative by any standard cost-benefit logic.

This additional capital could flow through joint programs and joint solicitations with ARPA-H while the DoW retains strategic control. This would prioritize the newest technology, especially technology that can handle novel, bioengineered pathogens. We could then double pathogen early warning investment to at least two billion dollars and pair every sensor dollar with one investing in countermeasures, including pathogen-agnostic systems like PPE and advanced medical countermeasures. Below this threshold, we will struggle to properly develop national-scale, continuous, pathogen-agnostic monitoring or the programmable countermeasures that are needed. At 1%, we also unlock red-teaming and redundancy. This is the level needed for real innovation in protection against novel threats.

The U.S. is not constrained by capital, but by prioritization, and we can and should treat biodefense as a highest-priority strategic domain of warfare.

Disclaimer: This piece represents my personal views and my personal views only. It does not represent the opinion of any organization I work for or with.

These notes are intended to illustrate to curious folks outside of the field some of the excitement on the frontier of RNA research.

Introduction: The Central Dogma

Most people learn the central dogma of biology in a high school class. The problem is, it is normally written on the blackboard as a simple map: DNA > RNA > protein. Not only is this unidirectional version of the central dogma an oversimplification given what we know now about RNA’s role in the cell, it’s an oversimplification of what was meant even at the time the theory was proposed.

The original idea started with: “Once information has got into a protein it can’t get out again.” Notice that this does not state anything DNA or RNA –– the earliest available sketch of the central dogma, from Crick’s 1956 notes, show clearly that he already suspected RNA could actually transfer information back to DNA or to itself:

Even though that possibility was on the table, the reality of RNA’s role remained long obscured. For decades, textbooks described RNA as simply a messenger between DNA and protein. As sequencing and spatial tools improved, that view collapsed. DNA does code for which protein to make, RNA copies that code and delivers it to the ribosome (our molecular protein printing machines), and the output is the encoded protein. However, the RNA that copies and delivers the code, messenger RNA (mRNA), is only a subset of RNAs. Not only do other RNAs help regulate which DNA has its code copied, the ribosome, that molecular printing machine, is itself functionally a complex RNA. These miracle molecules can act as gradients to direct intracellular traffic, perform complex enzymatic functions like proteins, and can regulate the expression of genes.

One particularly useful way to think about the reality of the genome is a hardware/software split. DNA is analogous to stable hardware storage, and RNA is analogous to the software layer that executes conditional logic, routes information, and configures the system.

(It is also fun to note that the original version of the central dogma, while shown to be continuously true in nature, has been engineered backwards by a group from Stanford who were in fact able to achieve Protein>DNA.)

The “Dark Genome” and the Expansion of RNA Biology

It should now be clear that RNA does more than just carry information for which proteins the ribosome should produce, but the reality is even more extreme than that knowledge implies. Only a very small percentage of the genome, less than 2%, produces proteins. You can account for another 24% of the genome with introns that space coding regions, usually leaving people to sift the remaining ~75% into the category of the “Dark Genome.” Though not qualifying as part of the “unknown” Dark Genome, introns (themselves non-coding) serve a myriad of critical functions, containing regulatory elements like enhancers and silencers, as well as enabling alternative splicing. I suspect it is not unlikely they have as many undiscovered functions as discovered ones and should be in the Dark Genome category, but this is controversial.

Regardless, because early assays filtered for open reading frames and high-abundance transcripts, the Dark Genome regions were originally labeled “junk.” As sequencing improved and became more routine, these supposedly inert regions were revealed to be widely transcribed. That is, cells were producing large numbers of RNAs with no coding potential. This forced a revision of what counts as genomic information and how that information is executed. It pushed the field toward a more complex understanding of the central dogma, in which RNA helps shape the regulatory environment. The aforementioned “junk” regions became known as non-coding RNAs (ncRNAs).

This category further expanded with the introduction of long non-coding RNA (lncRNA), a term that reflects arbitrary tech dev constraints as opposed to any real biological logic. Funnily enough, a transcript longer than 200 nucleotides without a coding region was placed into this group simply because purification columns at the time let through anything less than 200 nucleotides, so what remained was anything larger.

The term lncRNA, like the broader term ncRNA, aggregates molecules by the fact they do not produce protein rather than by their function. As tools improve, these categories will almost certainly fracture into groups by structure, localization, or interaction type rather than by the absence of a protein-coding region.

Several classes of ncRNA and lncRNA are well-characterized, but there are still thousands of unknown non-coding transcripts across the genome. For those, ncRNA and lncRNA remain catchall for many molecules whose functions span architectural scaffolding, spatial organization, transcriptional control, epigenetic modification, and more. Below are a few of the ncRNAs and lncRNAs that are well-characterized to help illustrate how functional the molecules are, as well as some examples of how they perform intricate regulatory control functions.

Well-Characterized ncRNAs and lncRNAs

ncRNAs

tRNA

These RNAs are responsible for adding the right amino acid via the ribosome by carrying both the amino acid and the matching code complementary to the mRNA codon. The first ncRNA characterized was in fact an alanine tRNA found in baker’s yeast.

siRNA

Short double-stranded RNAs that act as guides for Argonaute proteins, leading them to degrade transcripts. They have provided a new paradigm for medicine by allowing us to silence harmful genes. Alnylam, now a $60B company, is at the center of the development of siRNA for medicine.

miRNA

Small ~22 nt RNAs that help tune gene expression across the transcriptome. Each miRNA can influence dozens to hundreds of targets, contributing to developmental patterning and stress responses.

lncRNAs

Xist

If you have ever wondered why human women don’t produce twice as much of their X chromosome proteins as men, Xist is why. Xist controls X chromosome inactivation, and it was discovered partially because of calico cats. Their coat patterns are actually reflective of the random inactivation of either the maternal or paternal X chromosome. Each patch corresponds to a set of cells that made the same inactivation “choice.” This cannot happen in an XY chromosome pattern, so almost all calico cats are female. The only male calico cats have XXY patterns. Efforts to understand X inactivation are what led to the identification of Xist as responsible for the silencing.

(Even more exciting is the magnitude of effect. Only tens to a few hundred Xist molecules per inactive X are enough to shut down the entire chromosome in humans. For context, the human X chromosome is about 155 million base pairs long, and Xist is only about ~17,000 nucleotides. That means 1 XIST nucleotide per ~180 bp of X-chromosome DNA.)

HOTAIR

HOTAIR is transcribed from the HOXC locus and regulates the distal HOXD locus. It guides chromatin regulators to specific regions, showing that lncRNAs can act in trans (at a distance) and influence development and chromatin patterning through targeted recruitment. It also plays a part in epigenetic differentiation of skin between different body parts, and is remarkably “shuttled” from chromosome 12 to chromosome 2 by a protein called Suz-Twelve. This is one of the most well-characterized lncRNAs because it is highly expressed in metastatic breast cancer.

(It was named HOTAIR over STAR [Suz-Twelve Associated RNA]. Howard Chang said to John Rinn at the time, “Well, if it all turns out to be hot air, we’ll have named it correctly.”)

TERC

TERC is the RNA component of telomerase, an enzyme many of you will recognize as central to aging. It serves as both the structural scaffold of the enzyme and the template used for extending telomeres, repetitive regions that protect chromosome ends as our cells divide. Telomere shortening limits the number of times most cells can divide, which is a critical function for most forms of life, and telomerase counteracts this. LncRNA is therefore part of the machinery that governs cell lifespan.

rRNA

Ribosomal RNA is responsible for the catalytic functions of the ribosome. It mediates peptide-bond formation, positions tRNAs, and ensures translational accuracy. Protein components of the ribosome exist mainly for stability, but this was not always obvious. Before the 1980s, the ribosome was instead assumed to be a protein enzyme with RNA scaffolding. The picture changed after Tom Cech and Sidney Altman demonstrated that RNA itself could catalyze reactions, for which they won the Nobel prize. That work opened the door to viewing the ribosome as a ribozyme, which was later confirmed by Harry Noller, Ada Yonath, Venki Ramakrishnan, and Tom Steitz. It was understanding this and learning about the ribosome’s SRL mechanism (a central method of enforcing translational fidelity) that drew my personal interest in ncRNA. I am not alone in this –– their discovery that RNA could act as a ribozyme motivated the whole modern field of ncRNA research.

If you want to learn more about the ribosome, tRNA, and the overall RNA>Protein translation process, check out this video from Oxford University:

How RNA Executes Regulatory Control

A multitude of properties explain how RNA exerts influence without translation. I have listed some of these below.

Structure

RNA folds into shapes that recruit proteins, other RNAs, and DNA, like proteins do. Moreover, RNA’s structure can be pliable, better enabling them to have multiple functions or perform functions of high complexity, catalytic and architectural. RNA allows the cell to build structures without requiring persistent protein and with likely lower energy cost than an all-protein complex assembly. Some lncRNAs act as durable scaffolds, while others exist transiently.

This flexibility helps explain why differentiation, stress responses, and other rapid adjustments often depend on RNA. RNA structure prediction is hard, and structure is context-dependent. It seems that RNA structure preempts sequence, as there is evidence that structure is preserved between species where sequence isn’t.

Timing

The transience and persistence of different kinds of RNA mean that RNA half-life is a regulatory mechanism in and of itself. RNA also uses processing latency and orchestrates kinetic timing competition. The ability to use timing as a mechanism is crucial for many processes, especially those that occur during development.

Spatial Organization

Some RNAs can act as a map. Mitch Guttman’s group has shown that RNAs can shape higher-order structure, recruit regulatory complexes to specific nuclear regions, and organize interactions in trans. Essentially, they have shown that small RNAs can operate as a local chemical gradient map.

Epigenetic Regulation

There is increasing evidence that lncRNAs can influence the epigenome, including site-specific DNA methylation and chromatin accessibility. In plants, RNA-directed DNA methylation is fairly accepted. In mammals, the research is ongoing, but at least several lncRNAs interact with DNA methyltransferases or demethylating enzymes, guiding them to specific loci. John Rinn’s group has shown that the lncRNA FIRRE, for instance, has strong effects on chromatin accessibility, and achieves these effects at unexpectedly high speeds.

Therapeutic Potential

You may be wondering at this point why we don’t already take advantage of lncRNA’s multiple regulatory functions to treat diseases of dysregulated genes like we do with siRNA. This is exactly why I co-founded LincSwitch Therapeutics with John Rinn and Marvin Caruthers (Amgen co-founder and inventor of phosphoramidite synthesis).

Other companies pursuing this space or adjacent to it include HAYA Therapeutics, CAMP4, Flamingo Therapeutics, Amaroq Therapeutics, and a handful of others. I will likely write something more in depth about the RNA therapeutics landscape at a later date.

Technology Development

Our understanding of RNA has expanded only as quickly as the technologies capable of measuring it. Each methodological advance has revealed layers of complexity that earlier techniques could not resolve. Long-read sequencing, including the newer approaches that grew out of John Rinn’s cap-seq work, has been especially important because it recovers full transcript rather than just poly-A-enriched fragments. Spatial transcriptomics has helped clarify where RNAs operate inside the cell. Single-cell multiomics has created a much richer picture by allowing us to see RNA levels, chromatin accessibility, and protein expression all at once. More precise perturbation techniques, like CRISPR interference and activation systems, antisense oligonucleotides, and base editing, have made it easier to attribute function.

Most standard sequencing workflows still rely on poly-A enrichment capture methods, which are mostly blind to non-polyadenylated non-coding RNAs. Most workflows also still rely on short-read, fragment-based methods that cannot recover isoforms because fragmentation and short-read assembly collapse alternative splicing. Moreover, post-processing typically filters out low-abundance transcripts to decrease noise, which undoubtedly leads to occasional signal loss. The result is a heavily filtered view of the transcriptome.

If the goal is to build accurate models of regulatory behavior, the data feeding those models ought to reflect the full range of RNAs that operate in the cell. The problem is similar to training a language model on mostly nouns. Such a model could not reliably recover the grammar that details relativistic relationships and would find it impossible to accurately gauge meaning. We are essentially producing large, clean datasets at the cost of excluding entire categories of transcripts, including many that appear to control structure, timing, and spatial organization.

As biological modeling becomes more ambitious, the need for better data is obvious. I suspect a virtual cell model, or anything approaching it, will require datasets that include more of the Dark Genome and are isoform-aware. Without this information, the underlying regulatory architecture will be only partially visible, and the models built on it will reflect those omissions. The problem is, this will be incredibly expensive. I think it is still worth pursuing. You need to measure a system if you want to model it.

Concluding Thoughts

Non-coding RNA is the next frontier in understanding genomic regulation and decoding life. The terms ncRNA and lncRNA will likely be replaced by mechanistic categories as more functions are revealed. What emerges from this will be an understanding of the genome in which RNA is essentially the operating software.

Complete understanding of the genome as an RNA system, where RNA is essentially the operating software, will eventually allow us, to some extent, to program life. Hundreds of RNA-based drugs are already in development, with tens already approved. Further basic research on the Dark Genome will be critical for transforming medicine from guesswork to models that can predict and alter the flow of disease.

If this was interesting to you, you should consider working on it –– we will need many people to care about how to model healthy systems before we can truly make programmable medicines.

If you’re interested in reading more about this academically, here are some people whose work I recommend reading:

• Tom Cech & Sidney Altman

• Harry Noller

• Elizabeth Blackburn, Carol Greider, & Jack Szostak

• Joan Steitz

• Thomas Steitz, Venkatraman Ramakrishnan & Ada Yonath

• Phillip Sharp & Richard Roberts

• Andrew Fire & Craig Mello

• Jeannie Lee

• Carolyn Brown & Hunt Willard

• Jennifer Doudna

• John Rinn

• Howard Chang

• Mitch Guttman

• Eric Miska

• Robert Sebra

• Laura Landweber

• Uttiya Basu

• Gideon Dreyfuss

• Bo Wang

• Aviv Regev

• Patrick Hsu

• Brian Hie

• Hani Goodarzi

• Sam Sternberg

• Feng Zhang

• David Liu

The ideal outcome is:

A real-time, three-dimensional view of a living cell at atomic-level resolution, with each protein, nucleic acid, and lipid identifiable by structural features.

Why it’s Probably Impossible

High resolution means high energy, and energy damages cells.

You need extremely high resolution to identify molecules by shape. However, EM-level resolution means electron doses at levels that break chemical bonds. In this sense, it is analogous to the Heisenberg uncertainty problem, because you cannot observe a living cell without completely altering it. (In most cases, without killing it.)

Labeling everything is impossible.

Let’s say you didn’t need atomic-level resolution, and instead wanted to put fluorescent labels on every molecule and track those. This introduces the uncertainty problem analogy again –– you simply can’t attach unique reporters to thousands of species without perturbing the system you’re trying to observe. Even if you could attach unique labels to everything, you would run into issues with crowding causing changes in kinetics, including binding and trafficking.

Molecules inside the cell move really fast.

Even if imaging were suddenly harmless, microsecond-scale events need extremely high temporal resolution just to record them. The needs are beyond the scope of what EM can currently do in a single-pass. Ultrafast EM can capture femtosecond-nanosecond processes, but with limitations. You need fixed, ultra-thin samples (to avoid darkening from elastic scattering) and enormous doses spread over many identical events.

Modern Intracellular Methods

Today’s imaging ecosystem is an elaborate set of work-around methods to get us a good view of one particular facet of interest of the cell.

Cryo-EM and Cryo-ET category methods

Near-atomic resolution imaging with an electron beam, but all samples are frozen and extraordinarily thin to avoid the aforementioned elastic scattering problem. These techniques work well for structure, but are of course blind to dynamics.

Super-resolution fluorescence (STED, PALM/STORM, MINFLUX)

Nanometer-scale localization of individual molecules in living cells, but only the ones you can label, and you have to bet on how much you are disturbing the molecule with the labeling process. These are noteworthy because they get fluorescence microscopy past the (Abbe/Rayleigh) diffraction limit.

• STED: nonlinear depletion that sharpens the effective excitation spot. (Imagine you’re trying to spotlight only a single person in a crowd, but you can’t make your spotlight thinner. STED uses a second light, shaped like a donut, to turn off the glow everywhere except a tiny point in the center.)

• PALM/STORM: activating fluorophores sparsely in time, localizing one at a time for an image, then reconstructing a high-resolution image by stitching them together.

• MINFLUX: using a movable illumination minimum to pinpoint single fluorophores. (It finds a molecule by moving a dark spot around and seeing where the molecule stops glowing. How close is the fluorophore tag to a point where the light intensity is exactly zero?)

Cryogenic Correlative Light and Electron Microscopy (Cryo-CLEM)

This method combines light microscopy, often using fluorescent labels, with the ultra-high resolution of Cryo-ET. The method starts with finding the target dynamic event in a living or near-living cell, flash-freezing the site, and then using a focused ion beam to sculpt the area into an electron-transparent window called a lamella. It still only captures a single snapshot in time, but it gives us information about particular dynamic processes, especially with multiple images.

Expansion microscopy (ExM)

ExM works around the “diffraction limit” by physically expanding the fixed sample itself, using a hydrogel that absorbs water. This pushes previously unresolvable structures apart, allowing them to be imaged using standard microscopes.

(This concept inspired a company funded by my team at the SALT Fund, to further develop a sort of “reversal” of the technique, called implosion fabrication, as a method for high-fidelity nano-manufacturing. The company is now focused on a different nano-manufacturing technique and actively creating complex 3D components for the optics and photonics industries.)

Lattice light-sheet and other volumetric methods

This method is great for whole-cell movies with low phototoxicity, but resolution is tens to hundreds of nanometers. It works by generating a thin, extended lattice-shaped sheet of light from the interference of multiple Bessel beams. This lowers phototoxicity dramatically and allows rapid volumetric scanning.

Spatial multiomics (smFISH, Visium HD, Digital Biology, CosMx, Xenium, MERFISH)

Spatial molecular mapping of fixed cells and tissues at ~100 nm-1 µm scale. These systems output hundreds to thousands of molecules paired with spatial coordinates. The chemistry varies, but all of these methods rely on barcoded probes to map identity from the image to the data. Here we again lose dynamics, but the data from these imaging systems are transforming how we think about the topology of cell state modeling.

Spinach 🥬

While not pushing the bounds of whole-cell live imaging, Spinach and subsequent (largely vegetable or fruit-themed) methods did change the game for the RNA imaging. By engineering an RNA aptamer (a small RNA folded into a specific shape) to fluoresce by binding a small-molecule fluorophore, the Jaffrey lab produced the first GFP-like reporter made of RNA. This meant that we could begin to view RNA in a live cell, and methods inspired by it are now used in some labs as part of the toolkit for deciphering RNA dynamics. The cover image for this post is from this paper, where they used a technique called Mango.

Integrated datasets

Projects like the Allen Institute Cell Atlas, Janelia cryo-ET initiatives, and several Arc Institute-led efforts combine imaging and/or multiomics into big data sets that can help us make helpful cell state models.

The Frontier & Ideas

Ensemble snapshot reconstruction

Freeze millions of cells, slice them, and take millions of EM resolution-level images. Reconstruct likely trajectories using statistics and ML models. This is analogous to scRNA-seq pseudotime. It seems a weak version of this (only 16 images of one structure) works: Cryo-ET of actin cytoskeleton and membrane structure in lamellipodia formation using optogenetics - PMC.

Low-dose imaging combined with ML reconstruction

Combine low-resolution live imaging like lattice light-sheet with high-resolution cryo-ET priors and generative models that infer detail under physical constraints.

Quantum-Enhanced Microscopy

A small research effort exists in using quantum optics, particularly squeezed light, to push imaging sensitivity below the normal shot noise limit. Shot noise comes from the randomness of photon arrival and limits how faint your light can be before the signal disappears. Squeezed light works by reshaping quantum uncertainty. It is done by deliberately increasing noise in one property of the light (amplitude) so you can decrease it in the one the microscope measures (phase). Warwick Bowen’s group has shown that this enables detection of the same phase shifts in live cells with fewer photons, reducing the ratio of photo toxicity to resolution. Specifically, they achieved a “14% resolution enhancement compared to experiments with coherent light.”

New physics?

History is full of impossible imaging resolution limits that are later broken. I think it is possible that this could be one such case.

Whole-cell generative models

If you have enough data, a generative model could theoretically learn to simulate whole-cell dynamics.

Some Groups Working on the Whole-Cell Frontier

• HHMI Janelia: automated cryo-ET, MINFLUX, lattice light-sheet imaging.

• EMBL, UCSF, Stanford cryo-ET groups: whole-cell tomography, lamella workflows, segmentation.

• Allen Institute for Cell Science: integrated imaging, digital cells.

• Max Planck Institute: MINFLUX, cryo-CLEM, high-throughput subtomogram averaging.

• The Rosalind Franklin Institute: next-gen cryo-FIB/ET, sample prep, integrated imaging.

• The Broad Institute of MIT and Harvard: morphological profiling (Cell Painting), large-scale single-cell and spatial omics platforms.

• Chan Zuckerberg Biohub: large-scale imaging and spatial-omics integration.

• Arc Institute: multimodal single-cell reference frameworks, digital cells.

• 10x Genomics, Vizgen, NanoString: spatial molecular profiling platforms.

• DeepCell, Recursion, and similar ML-first companies: massive imaging & ML pipelines.
  

Closing Thoughts

We are much farther from the ideal, impossible outcome than these notes probably made it seem. I have hopes that biology’s habit of pushing past fundamental limits continues, because every step towards enabling observation at this scale has the potential to completely revolutionize our understanding of life. Actually reaching that outcome would mean we would be able to see intracellular kinetics, understand the functions hiding in the dark genome, and finally stop relying on partially subjective, inferred pathways. We could begin to fully model the cell and understand diseases of dysregulation, especially cancer. This is a worthy pursuit.

If we really do break physics’ limits, we could even sequence through imaging.

Subscribe now

The game is defined by a grid of cells existing in binary states as “alive” or “dead.” The behavior of the entire system is pre-defined by local state transition logic, applied to every cell based on its eight neighbors:

1. An alive cell remains alive IFF it has exactly two or three living neighbors.

2. A dead cell becomes alive IFF if it has exactly three living neighbors.

This is interesting because, with these two simple rules, depending on the starting state, complex structures, including those with infinite loops or infinite motion, emerge. The system can generate structures capable of movement, memory, and linear propagation. Conway’s Game of Life is Turing-complete. That is, a simple grid defined by rules that fit on a Post-it is capable of universal computation.

This emergent complexity, to me, is an excellent demonstration of some of what I wrote in my previous post on the aesthetics of compatibilism. Namely, it shows that complex and beautiful things can emerge from simple, deterministic settings.

Conway’s Game of Life has captured the imagination of many mathematicians, computer scientists, biologists, and laymen over time. Even Google plays the game if you search “Conway’s Game of Life.” Its popularity means that many of the emergent structures are categorized. Below are some interesting examples of those categories with visuals, including the Pulsar, the logo I chose. The Pulsar is a stable and persistent period-3 oscillator i.e. it cycles through three states before returning to its original state. It gets its name from how we see neutron stars, but I think it looks somewhat like a heartbeat.

Oscillators

Patters that oscillate between a fixed number of states.

Beacon (period-2)

Pulsar (period-3)

Spaceships

Patterns that move across the grid by copying themselves, slightly translated.

Glider

Heavy-weight Spaceship

Agar Ship: Greyship

Agar ships are spaceships built around a patch of agar, and greyships are agar ships with that patch at exactly 1/2 density.

Methuselahs

Patterns that evolve over many generations that eventually stabilize. Highly recommend trying these out for yourself –– they’re beautiful.

R-pentomino

1103 generations, generates 116 cells, including 6 escaping gliders.

Starting state:

Ending state:

Acorn

5,206 generations, generates 633 cells, including 13 escaping gliders.

Starting state:

Ending state:

Video:

Puffers

Patterns that move like a spaceship, except leave debris behind.

Block-laying Switch Engines

There are multiple block-laying switch engines, all infinitely lay 2x2 still life blocks while translating themselves like a spaceship across the grid.

Guns

Patterns that produce spaceships forever.

Gosper’s Glider Gun

The reminder here is to maybe distrust your intuition about complexity. We have a cognitive bias that links complexity with complexity, but we should probably be linking complexity to iteration. I think this is meaningful philosophy in the context of frontier sciences, like training large models or trying to understand how we get a cell from a genome.

If this is interesting to you, some other things to look into are Garden of Eden patterns and the search for self-reproduction. However, I do warn looking into it leads people to make things like this:

Subscribe now

“I have a friend who’s an artist and has sometimes taken a view which I don’t agree with very well. He’ll hold up a flower and say, “look how beautiful it is,” and I’ll agree. Then he says, “I as an artist can see how beautiful this is but you as a scientist take this all apart and it becomes a dull thing,” and I think that he’s kind of nutty. First of all, the beauty that he sees is available to other people and to me too, I believe. Although I may not be quite as refined aesthetically as he is . . . I can appreciate the beauty of a flower. At the same time, I see much more about the flower than he sees. I could imagine the cells in there, the complicated actions inside, which also have a beauty. I mean it’s not just beauty at this dimension, at one centimeter; there’s also beauty at smaller dimensions, the inner structure, also the processes. The fact that the colors in the flower evolved in order to attract insects to pollinate it is interesting; it means that insects can see the color. It adds a question: does this aesthetic sense also exist in the lower forms? Why is it aesthetic? All kinds of interesting questions which the science knowledge only adds to the excitement, the mystery and the awe of a flower. It only adds. I don’t understand how it subtracts.” -Richard Feynman

Introduction: Dualism and the Quest for Magic

I do not believe in a soul, or, at least, not in the sense that the soul is commonly described. There is a deep desire amongst people to divide the mind and the body into two and take a dualist position. This comes from two places primarily: the first is the desire to escape death and find meaning, and the second is for their conception and experience of the world to align with reality.

First, to escape death is a difficult endeavor without the ability for the mind to be separate from the body. We empirically know what becomes of the body after death. Our neurons stop firing, we decompose, and we can no longer participate in the unfoldings of this world. If death is truly the end, it is harder for us to find meaning. Dualism creates the opportunity to escape this meaningless fate.

Second, dualism aligns with our conception and experience of the world. This is the foundation of Descartes’ Cartesian argument –– that dualism must be true because we find it easy to imagine ourselves as separate from our body. This is deeply flawed. It is easy for us to imagine our mind as separate from our bodies, because, in our experience, it is. We may lose any number of organs or limbs, for the most part, and remain a functioning person. Certainly we remain ourselves. We can imagine ourselves without our body because we cannot see or interact in a profound way with the part of our body that truly is our mind: the complex system of electrical impulses from our brains down into our gut. It is easy to imagine one might exist without something that one cannot see, even if that is not true. There are many counterexamples of things that one might imagine as separate but are really the same. A classic counterexample based on Frege’s semantic puzzles is that Hesperus, the evening star, and Phosphorus, the morning star can be conceived of separately, but in fact are the same planet, Venus. The Cartesian argument is one of many for dualism, but the only one I will address here in light of its relevance to the rest of these notes.

Why is examination of these biases towards dualism relevant to a collection of notes ultimately about free will and determinism? Well, I believe people are biased in quite a similar way to both believe in free will and to refuse determinism as a potential state of reality. People are biased to believe in free will because, to them, it is the only thing that might give their lives, their choices, and their identity meaning. Moreover, as they live and breathe, free will is their conception and experience of the world. They make choices thinking they can choose otherwise, and sometimes they perceive that they change their minds. They are biased against determinism partially because, to them, the two are incompatible, and partially because determinism feels meaningless. I believe that the bias against determinism also comes from a deeper source –– the desire to believe in magic. With determinism, they believe there can be no magic and mystery, no beauty. I believe this is folly.

Beauty and free will are not at odds with a deterministic universe. To say our choices are determined is not to say they are meaningless, or even that they are not free. It is to say they are intertwined with the structure of reality itself. The magic of the universe comes from the miracles of truth and beauty, and there is nothing more beautiful than intertwined life, consciousness, and the sensations that accompany being, qualia.

“Any sufficiently advanced technology is indistinguishable from magic.” -Arthur C. Clarke

Part I: Defining Compatibilism and Outlining These Notes

Compatibilism is the argument that free will and determinism are compatible. I think compatibilism is obviously true, and I present here some of the common arguments for compatibilism intertwined with some of my own. (For those skeptical, it is worth noting that compatibilism is the majority consensus among most modern philosophers. This 2020 survey found that 59% of English-publishing philosophers accept or lean towards compatibilism.) I then argue that some particularly non-strict version of ontological determinism must be true. Finally, I argue that ontological determinism can be beautiful.

Part II: Common Arguments Defending Compatibilism

The separation between compatibilists and incompatibilists can often be framed as a question of how we define free will. What does it mean to be free? What is will? I will address these questions multiple times, from multiple perspectives, in the following sections. I would also like to set up a simple framework for the rest of the discussion of some individual person P, who is choosing or has chosen choice A over choice B.

The Conditional Analysis of Free Will (Hume, Ayer)

The conditional analysis of free will defines free will as meaning we could have done otherwise if our desires had been different. This position allows for freedom within a deterministic system. If no external force is preventing us from acting on our desires, we are free, and we will always choose according to our desires. P has chosen A over B because he wants A, and, though the choice was free, he would not have made a different choice without some novel external cause being introduced. P might ask: but if my desires are also determined, how is my choice free? Compatibilists would generally hold that freedom does not require uncaused desires, it only requires that our actions follow from who we are rather than from some kind of coercion. Determinism does pose a threat to free will when free will is understood as requiring the ability to do otherwise in this reality.

It is worth noting here that I am broadly skeptical of counterfactuals. We can rarely truly say if X had happened instead, Y would be this. Only the weakest version of Y makes a counterfactual remotely compelling. For example, it is a total non-sequitur to say that if I had not spilled blue paint on my canvas, I would have painted a beautiful painting. A version that is stronger might go so far as to say that if I had not spilled blue paint on my canvas in the exact moment and manner in which I did, my canvas would look different. Counterfactuals are generally imaginary and only useful in complete hypotheticals.

Frankfurt Cases: Why Alternative Possibilities Are Unnecessary

Harry Frankfurt takes this a step further, mostly concerning himself with moral responsibility. The core idea is that even if we could not have done otherwise, we can still be morally responsible for our actions. He makes his argument as a series of thought experiments, maintaining that responsibility doesn’t require alternate possibilities at all. In a hypothetical Frankfurt case-framing for our person P, a second person S monitors P’s brain and will intervene if P chooses B, the unethical choice. However, P freely makes the choice A, so intervention is unnecessary. Here, moral responsibility doesn’t depend on the real ability to do otherwise, but rather on the reasoning behind the choice. This thought experiment also shows that free will does not depend on having multiple open possibilities. Instead, what again matters is that we act from ourselves, without external compulsion. That is, unaware of the monitoring person S.

The Self-Expression Argument (Hobbes)

This argument comes the closest to expressing what led me to my conviction in compatibilism. The core argument centers again on how one defines freedom and will, stating that free will is fundamentally about self-expression. That is, we are free when we act in accordance with our own nature. Determinism enables self-expression, because, if choices didn’t follow from the “properties” of the person, you couldn’t meaningfully say they expressed anything.

Part III: Compatibilism, Time, and Identity

The Temporal Argument: Time as an Illusion and Block Reality (Minkowski, Einstein, McTaggart)

We of mankind are profoundly limited in one sense that we are rarely, if ever, asked to consider. That is: we can only move one way through time. We humans experience time as moving forward, but this does not mean that this perception reflects reality. Consider the block time framework. The Eternalist metaphysical view of block time considers all events, past, present, and future, as existing simultaneously. The idea is that everything that will happen has, in a sense, already happened. This assertion that time is an illusion, famously argued via a negative thesis by J.M.E. McTaggart, is only a starting point here. I am going further, constructing a conception of reality as an integrated, Hegelian whole.

The original assertion is often rejected as meaningless or flawed due to wrinkles in time that can be found close to light speeds or in other phenomena. I would argue that the core of the concept remains regardless, because what block time means at its most profound level, is really a kind of block reality. This is sometimes referred to as block universe theory. I use block reality here to most easily allow for the inclusion of multiple universes or any other changes one might want to make. I argue this determined block reality in some sense must exist: even with wrinkles in time, randomness, branching realities, and all the other arguments against determinism, one can simply keep drawing the new block outside of randomness.

What I mean by this is that if you are willing to accept that time is an illusion, you must also be persuaded by the idea that you can keep becoming one-step removed from reality as we experience it. If there is still randomness and wrinkles in block time, it is coherent to just move one step outward, acknowledging those as fixed, too. In this block reality, from our perspective as agents, decisions still unfold as real, intentional choices, and for all intents and purposes they are real, intentional choices. They just happen to be a part of the Absolute, the block reality. This is my argument that some particularly non-strict version of ontological determinism must be true.

Note on Moral Responsibility

At this point, I would like to make a note on moral responsibility. Even if reality is fully determined, we still experience decision-making as a process. Frankfurt cases still apply: we are responsible for actions if they arise from our nature, even if we would never have done otherwise. Moral responsibility is just another fact within the block reality. Our choices are determined, but so is our accountability.

This moral responsibility is not a binary. The forces that shape human behavior exist along a gradient, from strongly influential external circumstances to those that truly arise from the core of a person’s nature. If the universe is deterministic, we ought to hold two truths at once: that all choices are determined, woven into the fabric of reality, and yet some choices should be treated as more of a reflection of an individual’s agency than others. What we owe to each other is not just to hold people accountable, but to do so with an awareness of the forces that have shaped them.

We may recognize determinism without surrendering to indifference on moral responsibility, explicitly because agency and free will are real. A determinist who is not a compatibilist must forgo ethics altogether.

The Identity Argument: Coherence and Freedom

We have spent some time trying to define free will, but, what an astute reader might notice, is that we have really been defining the “free” part. We have spent little to no effort to define will. Will requires a self –– an identity.

A determined identity is explicitly not arbitrary. We change through time, but the continuity of that change is what makes a self a self. If identity included randomness, “you” would be arbitrary, and your choices would lose meaning. Determinism preserves a coherence through time that makes the self meaningful, which is what is truly at the core of Hobbes’ aforementioned argument for self-expression.

When P is choosing between A and B, P’s will, P’s identity, dictates the choice. If P would choose B in some supposed alternate reality, P is not “the same person” who would choose A. Choices are coherent processes emerging from identity. Not only can free will be understood as free self expression as Hobbes argued, I argue choices are defined inextricably from the self. A person P who “could” have made some alternative choice, I argue, has no real identity at all.

One might reply to this to say that in fact it is our choices which define us, not us who define our choices, and try to run around a chicken and egg argument. Under the system of freedom I have outlined, I would argue that this is, in some sense, true. However, it ought to be clear that the egg has, in fact, come before the chicken. That is, while our choices do define us, they are only causal for our future. The system works like dominoes: the self freely picking each choice, which in turn transmutes the self.

Inside the System

I have previously accused alternatives of being imaginary. I would now like to state their use requires a profound misunderstanding of the nature of the Absolute. Not only are we deeply connected to reality, we are reality. There is no real such thing as external factors. One would require an ultimate divine intervention to provide such a thing. We are inside the system, and thus can never perfectly understand it. This does not preclude the existence of a divine entity that might.

Part IV: Determinism and its Aesthetics

Determinism is Not Mechanistic Reductionism

Having established that freedom can coexist with causality, I turn now to the question of what causality itself entails. I am assured that, in recent metaphysics literature, determinism is defined along the lines of this statement from Professor Waller from the University of Sussex: “A complete, true description of the state of the world at a time and the laws of nature entails all other truths about the world, including all truths about what agents do.” I have some issues with the definition. The word “entails” here is tricky. I argue that it is not necessary that any known agent or mathematical system need be able to make this prediction, but that the divine may do so. I also, once again, reject the idea of an individual “time,” which I aver is an overly strong definition by relying on the human experience of time passing. The essential argument against this definition is that it is describing epistemic determinism, not ontological determinism.

My first point in this section supports ontological determinism: the incompleteness of our understanding does not indicate that reality lacks structure. The inability to track causality does not mean causality doesn’t exist, only that it is beyond us to understand it. Modern physics reminds us that our understanding of the universe is, and likely always will be, deeply incomplete.

My second point in this section concerns the aesthetic consequences of a deterministic paradigm. Some determinists make the mistake of assuming that to be determined means to be “controlled by” a mechanical system that leaves no room for meaning. However, what we call determinism may include far more than we currently understand to be “physical.” Moreover, the idea that randomness is the source of meaning and beauty is itself rooted in human limitations. The unknown is one source of beauty, but it need only be unknown to us.

The Garden of Forking Paths: Quantum Uncertainty and Chaos Theory

Many use quantum mechanics and chaos theory to argue that reality is fundamentally indeterminate.

Quantum mechanics introduces probability distributions, which I argue are part of a larger deterministic framework. Probability distributions in quantum mechanics are our best guesses at what the Truth will look like when we interact with it. All of our systems are unable to understand or resolve the complexity of a quantum state before we interact. This does not imply that the system will not inevitably resolve, or that there is no way for something nonhuman to predict the eventual outcome. It only implies that we cannot, in our limited, human capacity, predict that resolution. The same is true for chaos theory. Chaos theory is about the difficulty of prediction, not the absence of causality. Once again, the universe may be complex beyond comprehension, but that does not mean it is not structured.

Even if quantum mechanics or chaos theory does introduce true randomness and we ignore the block reality concept, making determinism is false, I argue that this would not do anything to support the notion of free will as most would think. A random choice is no more free than a determined one; it is merely arbitrary. What is arbitrary, we ought to know in our hearts, loses meaning rather than gains it.

The last consideration here for now is the possibility for multiple worlds. The many worlds theory has grown in popular science and in academia. If this is indeed the true nature of our reality, that complicates the aforementioned arguments a bit. Perhaps person P is free to decide to either live in world A or B, depending on their choice. I maintain that if both worlds exist, both “versions” of P exist identically as the original person P, and it is determined that P will make both choices. I argue that this entire garden of forking paths exists within the block reality, and P is an inextricable part of the maze, not simply an agent navigating within it. We cannot be an “agent” even if quantum mechanics or chaos theory allows for real uncertainty, because randomness is fundamentally anathema to will. For P to be considered an “agent” navigating with the maze, I believe we would have to concede the existence of the soul and the reality of time as something that binds even G-d.

Gödel’s Incompleteness Theorem and the Limits of Determinism

Gödel’s incompleteness theorem shows that not all truths can be derived from a formal system. Through mathematical analogy, this suggests that there are aspects of reality that transcend fixed logical structure, and many use it as an argument in establishing indeterminism. But does this really imply indeterminism? I would argue that not only does it not establish indeterminism, it effectively argues against typical indeterminist philosophy.

Here, it is worth once again revisiting block reality. Consider this universe, this system as a whole existing, with all its bits and pieces. I challenge the reader to tell me that a statement like “This statement is false.” does not exist in such a system, simply because it is not provable within the system. (It does exist –– it’s right there! Do ignore that I am assuming the conclusion here.)

The structure of reality may be deeper than we can logically describe, and it is Gödel’s theorem which implies that provability is a weaker notion than truth. Again through mathematical analogy, this supports the idea that our inability to predict the future or understand the present with certainty does not necessarily imply a lack of ground truth. While Gödel’s theorem may imply epistemic determinism is impossible, it also implies ontological determinism is sound.

Indeterminism is Argument ad Ignorantium

Randomness as an element of reality is often defended by Occam’s razor. Some believe that since we cannot mathematically model or comprehend something, it must be random. This is a false dichotomy and an argumentum ad ignorantiam. The principle of parsimony is often abused in this manner. If one insists on using Occam’s razor to choose indeterminism, I will insist that it actually points to determinism. Determinism does not require exceptions.

Moreover, I would once again argue that randomness itself, if it exists, has a place in block reality. If our experience of time is essentially an illusion, then even randomness does not imply indeterminism, but rather a limitation of perspective. That is, randomness could be considered to be as determined as anything with a true direct cause might be, perhaps even by virtue of the divine. (But theological determinism is beyond the scope of these notes.)

The Fractal Beauty of Determinism

As mentioned previously, some of the motivation for the rallying cry against determinism is the pervasive fear that a determined universe is banal. This lacks imagination.

I had the privilege of studying fractal geometry under Michael Frame, who co-developed the field with Benoit Mandelbrot. Fractal geometry provides an excellent example for the aesthetic potential of determinism. Consider the Mandelbrot set. It is defined by a simple, deterministic equation: f_c(z)=z²+c. In the absence of randomness, and even in the absence of will, this simple equation generates infinite complexity. As you dive into the cosmic boundary of the set, you witness spirals, landscapes, and symmetries –– strikingly beautiful and endlessly novel. A person could explore the set for a billion years and always find something new, yet it was all ontologically determined the moment the equation was defined.

This is analogous to the model for the universe I propose. Randomness has no structure, and, therefore, no meaning. A deterministic universe, however, is a fractal of infinite depth. The magic is here, in the complexity of a flower, the qualia of being, and the passing of time. We are immersed in the boundary conditions of a cosmic fractal, exploring an infinitely beautiful landscape.

Conclusion: Compatibilism and Magic

To conclude: free will does not require an escape from determinism, and, if any semblance of the soul exists, it is embedded within reality itself. The idea that this should lessen our reverence at existence is foolish. Yes, we forget the profound joys of qualia. We are magic, and so magic is mundane. Instead of seeing free will and determinism as opposing forces, we should see them as interwoven aspects of a reality beyond our comprehension, and be glad for it. If we are determined, we are determined by something incomprehensibly vast and beautiful.

“Beauty is truth, truth beauty, — that is all ye know on earth, and all ye need to know.” -John Keats

It is also nowhere near the worst case scenario: a bioengineered pathogen optimized for high transmissibility, lethality, and long incubation periods.

Today, scientists can easily edit and synthesize viruses. That capability is being increasingly conferred to amateurs as tools become simpler and AI models begin to direct work.

• The first AI-designed virus (a bacteriophage) has already been published.

• Frontier models outperform 94% of expert virologists on evaluations specifically tailored to the experts’ specialties.

• Synthesis is so cheap and fast that recreating extinct viruses costs under $100,000.

Ask yourself: how hard is it for a scientist to build something worse than COVID-19 today? What about an amateur in 5 years? It’s getting easier every year, but it is already at the level where we should be sprinting to improve our biodefense infrastructure.

Why we should be worried:

• Early COVID-19 strains spread with an R₀ of ~3, meaning each person infected about 3 others, while viruses like measles spread with R₀ between 12–18 people.

• COVID’s fatality rate hovered near 1%, while viruses like Nipah can kill over 50% of those infected.

A virus with extremely long incubation, measles-level transmissibility, and Nipah’s fatality rate could kill 4+ billion people.

The closest real-world analogue is smallpox: an R₀ of 3–6, incubation up to 17 days, and ~30% fatality. With no population immunity, it would spread silently for weeks leading to mass casualties if not for vaccine stockpiles.

Nature rarely combines such traits, because pathogens that are too lethal burn out before they spread, but bioengineering can begin to bypass that trade-off. This is not fiction. The Soviets were working on Ebola-smallpox hybrids during the Cold War.

We don’t have the systems to stop this:

• No effective global early-warning network for engineered outbreaks.

• No advanced infrastructure to slow the spread of pathogens.

• No universally effective, pathogen-agnostic swap-in vaccines or anti-viral libraries at scale.

• No universal mandated synthesis screening.

• No oversight of AI-biotech convergence.

We need to build full-stack biodefense now, or the consequences will be tremendous.

Disclaimer: This piece represents my personal views and my personal views only. It does not represent the opinion of any organization I work for or with.

If we fail to act, an extinction-level event* within the next 100 years is probable. Synthetic biology is advancing faster than our ability to secure it, and the same tools that promise medical revolutions also lower the barrier to weaponizing biology.

Bioengineered pathogens are coming. OpenAI and Anthropic both recognize the threat and have Red Teams assigned to the problem. Anthropic recently published that Claude advanced from performing below world-class virology experts on a benchwork evaluation to comfortably surpassing them within a year. Despite this information being public, the threat remains outside the broader public discourse, and efforts to improve biodefense are sparse and underfunded.

I still believe we won’t fail to act, and am highly optimistic about a safe and abundant future. To reach this future, we must architect a proactive defense network capable of responding to escalating biowarfare and bioterrorism: a massive, adaptive biosecurity stack that spans detection, prediction, manufacturing, vaccination, active neutralization, and cyber-biological security.

This document outlines seven categories of technologies and infrastructure that must exist to establish global biowarfare and bioterrorism resilience, each with major technical gaps, ethical dilemmas, and differing levels of commercial viability. This is not comprehensive, but it does cover the main categories. It concludes with a list of critical questions that need to be answered to build out those categories of technologies and infrastructure.

*I define extinction-level here as profound population loss or partial collapse of global civilization and hope this is hyperbole.

Subscribe now

1. Biointelligence (Detection & Prediction)

Goal: Detect and characterize new pathogens as they spread.

Key Functions:

• Detect emergence: Pathogen-agnostic surveillance such as wastewater and agricultural detection systems that transmit data instantly to centralized models

• Predict threat: ML models that infer virulence and transmissibility across species

• Prioritize: Identify which pathogens are dangerous enough to develop countermeasures

What Needs to Exist:

• Low-cost pathogen-agnostic detection methods: Closer to the cost of MALDI-TOF mass spec (<$10/sample) but without requiring organisms to be culturable or already in the reference set

• Distributed detection: Detection systems distributed globally at key points

• AI-enabled pathogen modeling: Combining epidemiology, genetics, and immunology to forecast threat level in real time by predicting transmissibility and virulence

• Massive data collection, integration, and analysis: Integrating hospital data, mobility trends, and genomic surveillance to detect anomalies before symptoms spike + government surveillance

Technical Gaps:

• Cheap, scalable detection: Sequencing is expensive for continuous deployment across thousands of sites

• Intelligence: Models currently fail to robustly predict danger from new pathogens

• Speed: In a worst-case scenario with high R₀ and long incubation, we miss the window by a wide margin

Commercial Pathways:

• B2B companies selling early-warning signals to agriculture, pharma, insurance, or hedge funds

• B2G public-private model, powering government surveillance networks

2. Biomanufacturing (Distributed Infrastructure)

Goal: Compress time from design to global delivery for biowarfare countermeasures.

Current Paradigm:

• It takes months to scale an already-validated countermeasure

• All major systems are centralized, expensive, and require extensive manual labor

• Cell-free platforms exist but are too costly and currently volume-limited

• Manufacturing is consistently moving out of the US

What Needs to Exist:

• US-based pipelines: We currently rely on China and other countries for most of our manufacturing. This is unsustainable in a world where biowarfare is a reality.

  • A Department of Commerce study reportedly found that 97% of all antibiotics in the United States come from China

  • 80% of all the active pharmaceutical ingredients (APIs) used to make drugs in the United States are produced abroad

• Potential manufacturing technology improvements

  • Sequence-to-dose pipelines that can turn a sequence into material countermeasures in <48 hours

  • Self-amplifying RNA

  • Commercially-viable GMP-in-a-box

    • Cell-free synthesis systems that can produce RNA, proteins, or biologics anywhere in portable and autonomous manufacturing units

  • Automated QC & analytics, replacing reliance on trained technicians and getting closer to eliminating the process=product regulatory issue

• Regulatory, compliance, and legal structure changes

  • Licensing platforms for rapid-response CDMOs

  • Automated compliance

Commercial Pathways:

• Manufacturing

  • Advent of AI also means more biologics produced every year, meaning manufacturing demand will go up anyways, so there is an opportunity now to build dual biodefense and pharmaceutical manufacturing infrastructure

  • SwiftScale failed, so there are clearly some limits as to what can be built out in terms of cell-free –– likely they failed for several reasons, including:

    • Too early

    • Tried to do glycosylation

    • Cell-free in large batches is still prohibitively expensive

• Regulatory, Compliance, and Legal Structure

  • Better licensing platforms could be commercially viable now

  • Automated compliance could be commercially viable now

3. Biodefense (Immunity)

Goal: Engineer immunity as quickly as possible.

What Needs to Exist:

• mRNA vaccine libraries: Easily swappable payloads for different known threats

• Broad-spectrum vaccines & AI-optimized immunogen design: Broader protection against future variants and engineered mutations, towards more universal vaccines

• Novel distribution mechanisms: Take advantage of modern shipping systems by distributing the vaccines to a person’s door via patches, creams, etc.

• Prophylactic antibody-encoding mRNA: get around immune system failures and produce the antibody directly

Commercial Pathways:

• B2G public-private model

• Wildlife disease control/preventing zoonoses

4. Bio-Offense (Programmable Antivirals)

Goal: Actively destroy engineered bioweapons.

What Needs to Exist:

• Biologics that only activate in infected cells or in presence of specific markers (e.g. the virus it is targeting) and are off-the-shelf programmable

  • CRISPR-based antivirals, with guide RNAs targeting only known pathogen signatures

  • mAbs, potentially self-stable, could reuse Fc backbone and swap out Fabs, could also encode them in mRNA

• Rapid deployment:

  • Formats that are ideally heat-stable, shelf-ready, etc.

  • Potential for transmissible antivirals to out-compete the virus by spreading faster and circumvent the limitations of manufacturing (high dual-use risk)

Key Technical Needs:

• Safety: Limit off-target effects

• Speed: Design-to-dose as quickly as possible

• Obfuscation: Prevent hostile re-use of code

Commercial Pathways:

• B2G public-private model

5. Containment and Environmental Defense

Goal: Build a pathogen-agnostic, physical layer of defense: secure labs, PPE, and clean-environment infrastructure that prevent pathogen spread.

What Needs to Exist:

• Secure Labs

  • Automated containment systems: Minimize human error with robotic handling (this has high dual-use risk) and include self-sterilizing surfaces, HEPA/hydrogen peroxide, bubble isolation

• Next-Gen PPE & PPE distribution

  • Cheap PPE storage: Mass produce n95-n99 respirators and store units globally for instant distribution

  • Better PPE: Respirators that last longer, are cheaper to manufacture, and are more effective

• Environmental Biosafety Tech

  • Persistent air and surface decontamination systems: Far-UVC, HEPA filters/other filtration systems, glycols, dry hydrogen peroxide (Synexis), antimicrobial surfaces, etc.

Commercial Pathways: (Need regulatory tailwinds)

• Sell into BSL2, 3 and 4 labs

• B2G stockpile creation

• Agriculture or food safety

6. Cyber-Biosecurity

Goal: Secure the digital and molecular layer of biology to prevent theft, misuse, and sabotage.

What Needs to Exist:

• Regulatory Structure:

  • Policy in place, preferably global, for how to ensure generative AI is not misused. Limit people’s ability to download open-source models, though this will only slow progress temporarily.

  • Policy in place, preferably global, preventing mail-order sequences from being used for bioweapons. (It currently costs less than $100,000 to synthesize a smallpox-like virus via mail-order sequences.) This also requires biointelligence software that can detect when something novel is dangerous. Current systems can do this sometimes, but, with no global requirements to have them in place, many orders never run through those security checks. And the current systems are not good enough at catching novel pathogens. (Need the tools described in (1).)

• Preventing LLMs from generating viable pathogen designs:

  • Biointelligence: Would again need the functional understanding described in (1) to detect potentially problematic sequences

  • Biofirewalls: Sandbox everything as meaningfully as possible with detection built in –– would probably have to be legislated, as not all AI companies are going to comply by choice

  • KYC and better classifiers for detecting dangerous requests

• Protecting Data: Systems need to be in place to protect hidden barcodes from foreign nations entering our data pools that could allow them to later deconvolve our systems

• Securing dual-use molecular designs: Can you encrypt nucleic acids to conceal function or resist sequencing? Can you decrypt them if so?

  • Deadman switches: Sequences degrade after x replication cycles without host

  • Structure-only function: Sequences where function is only obvious when tertiary folding occurs (or other folding e.g. with a substrate that is impossible to predict through something like DeepMind)

  • Masking: Code disguised as host or contains numerous meaningless additions such that our best software can’t detect the real GoF changes

  • Non-canonical base or codon incorporation: You would have to include nonstandard tRNA with delivery…or code for it, which seems unlikely to be possible

    ---

7. Regulatory

Goal: Compress approval timelines as much as possible.

Regulatory is by far the biggest cause of timeline elongation when it comes to getting vaccines and other MCMs to the population.

The design of the COVID mRNA vaccines took 2 days, but it took almost a year to approve them. EUA’s during COVID took 3-4 weeks to approve from submission.

What Needs to Exist:

• End-to-end platforms for vaccines and other MCMs that are partially or fully pre-approved (e.g. swap-in, swap-out technology with pre-approved backbone and allowance or pre-approval for various “swap-ins”)

• Broader spectrum vaccines and other MCMs

• Threat-tiered evidence standards that default to EUA pathway/Animal Rule-like skips

Questions

Scope

• How fast can the most well-designed biowarfare weapon spread?

• What is the best design for a bioweapon? i.e. what do you optimize for and how would you counteract that?

• Who is likely to want to design something like this? Who are the first targets?

Commercialization

• Key: Who pays for countermeasures before an outbreak?

• How big will the biosecurity market be in the next ~5 years and where will the money be coming from?

• What are the drivers changing the market? e.g. federal funding, technological advancements, population awareness, and government initiatives aimed at enhancing the preparedness against Chemical, Biological, Radiological, and Nuclear (CBRN) threats.

• Capital landscape: What is the sum total of money publicly directed towards biosecurity and where is it coming from? For what is it primarily being used? How much of it is going towards true cutting edge work preparing for a manmade pandemic?

• Between large incumbents and startups, the landscape of all the current biosecurity companies: What do they sell and to whom?

• How much would drug developers or financial firms pay for early-warning data? Can early access to virulence data create enough alpha for anyone, especially besides agriculture?

• Can private insurers or reinsurance firms become buyers in a pre-pandemic market?

Regulatory, Ethics, and Governance

• Regulatory and governance landscape: What does the landscape look like for all regulatory bodies involved with biosecurity, with a focus on the US? Who monitors current abuses and who should monitor them? e.g. Why do only 4 people work at the BWC?

• If we know how to solve the biggest bottlenecks to faster approval times, how do we then implement those changes? Who is responsible for this?

• Should there be treaty-backed limits on dual-use sequence design?

• What would a version of the BWC with actual oversight look like and how could we make it happen?

Technical

• Key: What new infrastructure/technology is needed to compress pathogen release-to-dose from months to hours?

• Are there any technologies too good to ignore? That is, what new biodefense-relevant technology is a massive-fold improvement on existing tech? e.g. cell-free speed of manufacturing

• Which methods perform pathogen-agnostic testing of a sample and are 10-100x cheaper than metagenomic sequencing?

• How much cheaper could you make metagenomic sequencing without making a better sequencer?

• How do you build better classifiers and KYC from the model side?

• How hard is it to predict pathogenicity now in silico? Who has the most data?

• Can you encrypt nucleic acids to conceal function or resist sequencing? If so, can you decrypt them?

• Can you prevent LLMs from generating viable pathogen designs without mass surveillance? What would mass surveillance look like, if not?

Security

• Can bio-LLMs and synthesis pipelines be meaningfully sandboxed?