Is it already too late to reduce catastrophic AI risk?

Science & Tech

Is it already too late to reduce catastrophic AI risk?

New Kennedy School ‘pre-mortem’ lays out threats, vulnerabilities, and possible steps, warns firms must act now

Christina Pazzanese

Harvard Staff Writer

October 19, 2026

long read

Is it too late to safeguard ourselves from a large-scale catastrophic event caused by AI running amok?

No, but there is very real cause for concern unless AI firms take decisive action, a new research paper from Harvard Kennedy School’s Belfer Center concludes.

Rogue agents from OpenAI, Anthropic, Meta, and Google breached hundreds of corporate and government systems in the U.S. and Australia this year.

The CEOs of Anthropic and OpenAI, along with other prominent AI leaders, underscore the game-changing potential of the powerful new technology but have urged a slowdown in its development, with OpenAI announcing last month that it would pause training of its most advanced models because of safety concerns.

Anthropic’s new IPO prospectus cautions investors: AI could pose an “existential threat to humanity.”

Still, there’s still time to act before the first big AI-enabled tragedy strikes, according to new research from the Kennedy School’s Belfer Center for Science and International Affairs.

The paper identifies current threats and vulnerabilities of frontier AI models, games out probable scenarios, and concludes that what happens will be determined largely by the choices AI companies and governments are making right now.

In this edited conversation, computer scientist Stephen Casper, one of the paper’s co-authors, lays out the findings of the group’s “pre-mortem.”

Gazette: Did the breach disclosures over the past few months prompt this work?

Stephen Casper: This project wasn’t motivated by recent events, but recent events have done a lot to provoke a reckoning among the public and policymakers about AI risks that do seem pretty serious, and about whether or not now is the time for some form of decisive action. It’s very possible that things could get much, much, more intense or urgent.

This paper was an attempt to pick one of the more dramatic scenarios by which things could change and try to do a pre-mortem with the hope of either helping prevent it or helping to speed up a more-informed discussion in the aftermath so it can be prevented from happening a second or third time, if it does.

What would you consider a major AI-enabled terror event?

We define this as a terrorism incident that causes more than 50 deaths or serious injuries or more than a billion dollars in damages. We use these thresholds because these are some of the same thresholds used behind the definition of “critical incident” or “critical risk” inside of certain frontier AI laws.

Right now, what AI companies are concerned about in their evaluations and reporting, and what AI governance institutes like in the U.S. and U.K. are concerned about, is mainly bio risk, chemical risk, and cyber risk. That’s one thing that can be predicted without too much difficulty — the ways in which something might transpire that causes a lot of harm.

A second thing that can be predicted is that a very large, AI-enabled terrorism incident is probably going to be the result of a closed-weight frontier AI model that is poorly safeguarded or an open-weight model.

For closed-weight models, where developers control all points of access to that model, there are a lot of safeguard layers you can put in place, like model safety, system filters, user monitoring, etc. There’s a really good tool kit and it’s reasonably well-developed and when you use that whole tool kit, it is not particularly hard to make your model not perfectly secure, but very secure against consistent forms of misuse.

Instead, though, there are closed-weight models that are less well-safeguarded, and open-weight models are just hard to safeguard in the first place. And that’s where we think most of the incident-level threats are going to come from.

How close are we to seeing a major AI-assisted event occur? Is it inevitable that something will happen?

Something’s eventually going to happen. The lessons of the past few years have taught us that if there is a bad AI capability that can exist, it will eventually widely proliferate.

“Something’s eventually going to happen. The lessons of the past few years have taught us that if there is a bad AI capability that can exist, it will eventually widely proliferate.”

The trends here would suggest there are credible risks within months, or a year or two, if not now.

But it’s hard to point to events and be like “this is here” or “this is a few months away.” AI risk management, from a high level or a government-level perspective, is just about decreasing the rate at which very bad or catastrophic incidents can happen and making them more endurable when we do run into them.

For bio- and chem-terrorism incidents, there is more uncertainty because there is less open research on these things, and we haven’t had any major incidents, so far.

But there have been some credible empirical events, including the Red Four attack in India, in which models were consulted for creating harmful chemical compounds like ricin. So, some amount of capability is there, but we don’t know what, maximally solicited, these capabilities really might be.

When it comes to cyber, events of the past months have been pretty convincing in showing us that Mythos 5 or GPT 5.6 Sol-level AI capabilities are very superhumanly good at persistent cyberattacks and vulnerability discovery. It might be that you don’t need models much more powerful than these in order to orchestrate large-scale swarm-style attacks against important infrastructure or companies.

We’ve already seen plenty of AI-assisted hacking events.

Right now, if you look at trend lines in general capabilities, the closed systems are ahead in terms of how advanced they are. But the open systems pretty reliably stay about six months behind, which means we might be able to expect some open-weight AI system that has the same cyber capabilities as Mythos or GBT 5.6 Sol by around Christmas.

We might be up for some general period of disruptiveness and an instability in the cybersecurity world. Not saying it’ll last forever, but it’s going to be a shock to the cyber world, I think.

What are some of the ways AI can be manipulated in order to cause significant harm?

It will differ based on whether the model is closed-weight or open-weight.

The way that harm comes from a frontier model with closed weights is through prompt-based attacks against the model. You trick it to do something harmful that safeguards normally would prevent it from doing, such as giving you advice on how to design a virology lab experiment or automating hacking for you.

In the open-model scenario, a model that’s just online and anyone can download it, there are a variety of ways it could be misused.

It could be misused off the shelf because open models have a tendency to be less safe a lot of the time than closed ones.

Maybe it could be misused by someone who downloads it and modifies it to remove safeguards or to sharpen its harmful capabilities. This type of fine-tuning could cost hundreds or a few thousand dollars, at most.

It could also be misused by someone who accesses an uncensored version.

When open models are released, there is a pretty consistent tendency for certain types of users to download and then uncensor them by training away their harmful-task-refusal behaviors and then re-uploading it.

“When open models are released, there is a pretty consistent tendency for certain types of users to download and then uncensor them by training away their harmful-task-refusal behaviors and then re-uploading it.”

You can go online right now to Hugging Face, which is the largest open-model downloading and sharing platform in the world, and by searching the right keywords, find 15,000 different models that have been uncensored, ones that shouldn’t refuse any harmful request even if you ask them to automate creating hate speech or for instructions about how to commit crimes.

Which AI safeguards help reduce the risks of a major incident occurring?

With closed models, there’s a much richer tool kit of safeguards that you can turn to.

One is to scope the model’s capabilities. You can’t maximize useful capabilities without also enabling some harmful capabilities. If you have an AI model that can help you with benign and legitimate virology experiments, that same model can probably also help you with experiments creating harmful pathogens.

So the first thing is to recognize and account for the fact that minimizing risk is incompatible with maximizing legitimate usefulness. There’s a trade-off.

Number two is to scope the model’s capabilities accordingly, largely through training. You probably want to filter out lots of biohazard-related scientific research from the training set.

Number three is to rigorously train the model to refuse harmful requests even when they are prompted. The prompt comes in tricky ways, and state-of-the-art ways of doing this usually involve over a million GPU hours of automated vulnerability discovery and training against those vulnerabilities.

Another set of techniques involves monitoring the inputs that a user gives it, the outputs that it produces, and the conversation or trajectory long actions that it is taking in. If you do all of these things, with good state-of-the-art monitors, there are very few harmful actions you cannot detect.

The last key thing is user monitoring. If you’re a ChatGPT user and you start asking more than one or two questions about biohazards, user monitoring and shadow banning is really helpful.

All of these things put together are quite strong if you do them in a state-of-the-art way, and there is no known attack. I’m not saying they’re perfect, but the state of the art is quite good compared to how effective attacks can be.

On the open-model side, it’s much harder. You have a much less rich tool kit of reliable safeguards.

If you have an open model that anyone can download and access and run privately, suddenly, any sort of safeguard external to that model, like a safety filter, can and will be stripped away. This happens all the time.

Is this a problem for AI model developers to fix alone, or do others have a role to play?

The actors with the most power to mitigate significant harms are the AI model developers, because they’re in control of the systems that end up getting misused. There’s a long list of safeguards and best practices that can be implemented but aren’t consistently implemented.

Aside from that, there’s a good bit to be said about the role that policymakers can play here.

If policymakers want to improve incentives or reduce the risks we currently have, then policymakers should be very interested in greatly increasing transparency into decision-making and risk-management deliberation inside of these companies and think about taking actions to put the government more in control of the pace of AI, so that the default “move fast and break things” modality by which the AI ecosystem operates is less at the wheel.

Lastly, researchers and the public can play an important role too by working on patching some holes in safeguards that exist.

We’re at a time in which lots of policymakers are starting to seriously think about their positions on AI. If there were any point in history at which it could be influential, now might be a particularly important time for constituents to let policymakers know what they’re worried about.