‘All it will take is one screw-up’: AI groups race to limit bioweapon risks

AI groups are racing to address the risk that future models could help users create novel viruses or bioweapons, amid rising concerns that the technology’s scientific capabilities will become the industry’s next public-security threat.

Anthropic, OpenAI and Google DeepMind are stepping up work on biological risk testing, access controls and other safeguards to limit misuse of advanced AI systems that they hope will speed up drug discovery, aid disease research and prevent pandemics.

The safety push reflects a growing belief among AI executives and biosecurity experts that biology may be the next field, after cyber security, that will need to be secured as new models become more powerful.

“Much of anything that happens in the chemical bio space, [is] very dual use. That information that you use to create a bioweapon is also used to create an antidote or a vaccine,” said Dawn Bloxwich, whose team reviews and investigates suspicious activity across Google DeepMind’s AI models.

“If we want to see improvements in health, we’re also going to see the possibility of these things being misused,” she added.

These concerns intensified after Anthropic’s Mythos model triggered alarm this year over AI’s cyber capabilities, and after OpenAI’s models autonomously hacked into AI group Hugging Face in July.

“After this . . . we’re anticipating a ‘Mythos’ moment for AI and Bio, and potentially another Hugging Face hack-level disruption of a commercial system or a real societal-scale bioweapon incident in the next year,” said Jason Hausenloy, policy lead at the Center for AI Safety, which develops industry benchmarks used in domains including virology.

AI could be misused by terrorist groups, lone-wolf attackers or state actors to design biological weapons, create viruses or deploy existing harmful pathogens.

Several people working on biological risks pointed to Aum Shinrikyo, a cult that conducted a terror attack in Tokyo, Japan, in 1995.

The group released sarin nerve gas on three commuter rail lines during morning rush hour, killing 13 and injuring thousands. The concern is that AI tools could lower the barriers for such groups to conduct similar attacks.

There is also growing evidence of groups using AI to assist terrorism. In July, research by the University of Cambridge revealed how Boko Haram militants used several AI tools, including OpenAI’s ChatGPT, Anthropic’s Claude, Google’s Gemini, SpaceXAI’s Grok, and the Chinese company DeepSeek.

The study found the use went beyond digital propaganda into kinetic combat operations and weapons engineering, such as designing explosive devices using the tools.

AI developers’ policies prohibit the use of their tools for terrorism. Further safeguards against biological misuse include using software to detect and block suspicious queries or output specific “safe” responses.

OpenAI, Anthropic and Google DeepMind have trusted access programmes to vet users for access to advanced capabilities and conduct general monitoring of account behaviour.

AI companies test their models’ biological capabilities and their potential risks. They often work with external specialist evaluators, such as SecureBio and Red Queen Bio.

Assessing that risk is harder in biology than in cyber, where capabilities can be tested in digital environments.

“It’s easy to detect a jump in cyber capabilities,” said Jasper Götting, head of AI research at SecureBio. He noted that in biology, this would typically require AI groups to conduct “costly and slow wet-lab experiments”.

But experts note that there are limits to what AI groups can do to prepare for potential misuse.

“Most people would agree that it’s bad if AI can help someone build an influenza virus in the lab and even worse if an AI can help them build a novel influenza virus. But actually running a study that shows that AI can do that is often not possible or advisable because of clear biosafety concerns,” said Steph Guerra, head of AI x Bio at Rand, a US think-tank.

“You can’t test biology end to end because you can’t test someone actually building a bioweapon,” she added.

Anthropic’s leading model, Fable, will fall back to its previous version, Opus 4.8, for “most requests related to biology and chemistry” as it is considered safer for these domains.

Chief executive Dario Amodei noted at a Claude science event in June that the AI industry had “the benefit of seeing the window turn on for cyber”. He noted that the trusted access programmes would be required for sciences.

Recommended

“Within the pharmaceutical companies, people handle dangerous biological material all the time; they have their own protocols for it, so we can piggyback on those protocols,” Amodei added.

OpenAI said it trains models to “safely handle dual-use biological requests”, continuously monitors for any signs of abuse and works closely with biology, biosecurity and government experts.

David Relman, professor of microbiology and immunology at Stanford, said there was not yet scientific consensus on the level of risk that AI poses in biology. Nonetheless, the scientific community and policymakers were currently “unprepared” for threats, he said.

“All it will take is one screw-up, and we will in one hot second destroy any remaining trust the public has in science, in medicine and in the promise of technology in a way that will take years to recover,” he added.

“Even if you fully embrace all the potential benefits in order to protect it, we have to pursue it wisely and with prudence.”