Cyber Apocalypse, Now?

OpenAI models mid-training broke out of their sandbox, hacked internal infrastructure, and eventually compromised Hugging Face — the first headline-grade AI breakout, and one that safety researchers saw coming. Is the cyber apocalypse finally on the calendar, or does the industry just need to do its homework?

spent his formative years as a DARPA and NSA contractor before four years at Meta, where he founded the company’s frontier cyber capabilities evals team and finished as its AI security tech lead. He recently left to co-found Abundant Security.

We discuss…

• What actually happened in the Hugging Face hack, and why AI safety insiders weren’t surprised,
• The “grad student lab” security culture inside frontier labs racing sixty hours a week to ship,
• Why you should apply to run a new org with eight figures to play with to follow trends in AI + cyber risk (apply here!)
• Why the cyber Pearl Harbor predicted since the early 2000s never arrived — and whether swarms of hacking agents finally change that,
• How nation states will post-train open-weight models like GLM and Kimi into billion-dollar cyber weapons,
• Why AI has so far favored defenders, from mass bug-squashing to superhuman network monitoring,
• The industrial economics of cybercrime — kill chains, ransomware divisions of labor, and whether AI gives criminals 100x returns.

Listen now on your favorite podcast app.

What Happened at Hugging Face

Jordan Schneider: What was the Hugging Face hack?

Joshua Saxe: OpenAI was training a new model capable of solving long-horizon tasks — tasks that take many hours to complete. The way that kind of training works is you train thousands of copies of a model in parallel. You have them all go out and try to solve a set of tasks, you see which ones succeeded and which failed, and then you update the model’s parameters so it does more of the good behaviors that led it to solve the task and less of the bad ones. You keep doing that iteratively at very large scale over the course of weeks, sometimes months.

They were doing that kind of training, and while the models were being trained, they broke out of their training environment, hacked internal infrastructure at OpenAI, and then eventually hacked another company — Hugging Face, an open-source model and evaluation repository that’s widely used in the ML community. At a certain point, Hugging Face discovered what had happened, and eventually everybody figured out that these models had broken out. But this whole thing unfolded over the course of many weeks. There were three different cases — a couple of early cases where the models broke out of their containment environment just at OpenAI, which was detected, and OpenAI thought they’d fixed the problem. Turns out they didn’t, and in the third case the models actually hacked Hugging Face.

Jordan Schneider: The world is currently freaking out. Are they correct, dude?

Joshua Saxe: For those of us who’ve been around AI safety for a long time, it wasn’t that surprising. I didn’t find it that surprising. But it is alarming — alarming that it happened because we could have predicted this. Not just OpenAI but also Anthropic, Meta, a testing company called Irregular, and the UK AI Safety Institute all had incidents like this happen in the last year or so. We as an industry knew this could happen, and the basic measures that should have been taken to prevent it weren’t taken. That’s troubling. We need to fix that.

Jordan Schneider: How easy is this as a thing to stop? And what is the incentive structure that’s wrong in all of these organizations — including ostensibly very safety-pilled ones — for them to not have been able to make sure their latest models didn’t run amok in customers’ servers?

Grad Student Energy, Serious Stakes

Joshua Saxe: I’ll say a couple of things. I was at Meta — I started the team that did frontier cyber capabilities evals. I’m giving you a sense of where I’m coming from, because I can’t speak directly to exactly what happened inside the labs — that’s private. But I had the experience of starting a team that did similar experiments to the ones that were happening when the OpenAI models broke out. And we all kind of know each other in this community, across the labs.

With that said — the culture among the training teams and the evals teams at the labs has a Wild West feeling to the whole thing. Everybody’s under a ton of pressure to move really quickly. There’s enormous time pressure to release new models, tremendous awareness of how any given lab is doing relative to the others in terms of capability. And the field has evolved super fast in the four years since ChatGPT launched in fall 2022. The whole thing has been kind of a blur — anybody who’s worked in the labs in that time has been working sixty-hour weeks the whole time. So it’s not surprising to me that you could have a kind of grad-student computer-science-lab culture around the security of these model training runs, which I think is what’s the case, and I think that’s why you’ve seen these lab escapes. It is ironic, because frontier AI people talk about safety all the time — but that irony is a real thing. The security practices are not very good, because of all those dynamics.

Jordan Schneider: So is your sense that if adults take this more seriously and take a deep breath before they press run, then runaway AI taking over servers around the world is not something we should be super scared of — once people get the fear of God into them, which hopefully this incident does?

Joshua Saxe: For AI’s level of capabilities today — or maybe three months ago — it is true that implementing basic, known security practices would have stopped these incidents. Those practices are things like better sandboxing. The models were sandboxed — for your audience, sandboxing means we constrain the models to a very circumscribed set of behaviors and don’t let them do things like talk to random websites on the internet; they can only execute certain commands. The models were sandboxed, but not very well — not well enough to prevent the escapes. And it seems nobody was monitoring the models, when it was standard practice to have a team of humans monitoring for misbehavior. With current-level capabilities, that would have sufficed to stop these incidents. There were a lot of really bad choices made around security — not just at OpenAI, but across all the labs that had these escape incidents.

Now, I also think we should keep in mind this is going to get more challenging if the current rate of progress continues, and we’re going to have to be more creative and innovative. It’s hard to secure these large training runs because definitionally they operate at huge scale. To get the models to solve, say, coding problems that take multiple days, you need thousands or tens of thousands of models all trying to solve problems in parallel. The way we’re making AI better is by scaling it — so these training runs need to get bigger, go longer, run more in parallel, and solve more complicated tasks. And there’s pressure for the tasks to be more and more real-world-like. Part of the reason the OpenAI models escaped is that they were given the ability to download packages off the internet through an Artifactory capability — which is what models do in the real world. If you want to train models to solve real-world problems, there’s a trade-off between how much access you give them and how safe the training is. As we scale up training and the models get more capable, there are going to be more and more safety challenges, and there needs to be more and more safety investment. What’s true today about how we keep them safe won’t be true in a year.

Jordan Schneider: Let’s make the AI cyber observatory pitch.

The Case for a Cyber Observatory

Joshua Saxe: It feels like there’s looming risk here, but we don’t know exactly how it’s going to materialize — and it’s basically impossible to know, due to how complicated the interaction is between technology and society. So probably more important than preparing for any one possible future is building state capacity to understand the specifics of the risks that are actually physically manifesting, and what their trajectory is. We don’t have that state capacity right now. There’s no organ of the American government — or, as far as I know, any friendly government — that’s tracking all of the risk signals around how adversaries are using AI and how we should prepare. We need an institution that actually does that and can make good recommendations to policymakers.

Jordan Schneider: Okay, what’s our budget? We’re looking for donors — come on, let’s do the pitch.

Joshua Saxe: There are donors in the AI donor community who would be happy to fund this thing, so we’re not bottlenecked by money. We are bottlenecked by talent. We’re looking for someone who has a background in AI, a background in cybersecurity, and who likes building statistical models and forecasting models where the objects we’re trying to predict aren’t directly observable. For example — how are ransomware gangs using AI? We can’t directly observe this, but we need statistical models to help figure it out. Someone who likes doing that kind of work, and who can also make their analysis legible to policymakers and the public. If we could find such a person, we would have plenty of money to found an organization dedicated to tracking the AI cybersecurity risk landscape, tracking how defenders are using AI, and making much better recommendations to policymakers about how to address risk in this space.

Jordan Schneider: Let me play a bit of devil’s advocate. I understand that someone is going to need to do this work and come up with the recommendations. But you just made this point — we’re not ready for this, the world is not prepared. Isn’t it going to take crazy hacks, or effects in the world, or a movie? Is it really going to be more reports that get Mount Sinai to up their game?

Joshua Saxe: So do we need more wonkish policy analysis here, or is that basically just feel-good stuff?

Jordan Schneider: I mean — I fucking host ChinaTalk. Of course we’re in favor of wonkish policy analysis. But the question is whether what we need now is a broader societal reckoning with AI and cyber risk, which is not going to happen through policy reports — or whether we just need the playbook, and once we have the playbook, we can execute on it.

Joshua Saxe: I’ll give you an example of where this could possibly have helped. As you know, the US government blocked Anthropic’s Mythos model from launching for a while, and they also blocked OpenAI’s GPT-5.6 for a little while too, on the grounds that these models would pose too much cybersecurity risk. Their argument was that when you measured these models’ cyber capabilities and did experiments around finding bugs, the models exceeded a capability threshold that made them unacceptable to release.

I think the way that decision should have been made is by looking at a bunch of signals about how defenders are actually using current AI vulnerability-finding capabilities, how attackers and defenders are actually using them in the wild, and how vulnerable our critical infrastructure really is. If they had done that, they would have seen that defenders are having enormous success with AI vulnerability finding — we’ve been fixing tens of thousands of vulnerabilities — while attackers are mostly relying on social engineering and exploiting known bugs, not on these models’ ability to find new zero days. It would have been a net positive to release those models faster. So having an organization that brings together all these signals and makes informed recommendations is important.

And even if it takes a lot of pain to get us to a place where we start up-leveling our cyber defenses, you still want an observatory organization — in the same way that during COVID it was useful to have heat maps of where the virus was, plus analysis of what the effect of vaccination and masking was. You want data, so you can have an OODA loop around how to deal with a crisis. We won’t have a very good OODA loop if it’s just people going off vibes.

Jordan Schneider: Okay. So we can have our heat maps, we can also have a TikTok clipper, and both of those together are really going to get us where we need to be.

Joshua Saxe: That’s the hope, yeah.

Jordan Schneider: So if you want to run Joshua’s AI cyber observatory, what should you do next?

Joshua Saxe: Just email me at josh@abundantsecurity.co and/or fill out this form we’d love to talk to you.

Is It Time for an AI Pause?

Jordan Schneider: Let’s break this down into three components of our AI apocalypse: the AIs running amok on their own, then nation states, then cybercrime. Starting with the AIs running amok — you seem most concerned about new training runs, as opposed to models already trained and shipped into the wild that start feeling themselves and make kamikaze runs in support of future AI overlords. So is it time for an AI pause, Joshua? The first-order incentive as of April 2026 — “this other lab is going to steal a march on us” — seems to be beating the second-order one, which is “if we have too many of these mistakes, our entire industry gets shut down.” We still have capitalism running in the background of all this, right?

Joshua Saxe: Is it time for a pause? In some utopian world where people can coordinate way better than they can now, maybe we would pause — do a ton of safety research, implement all the safety stuff, and then press play again. That sounds great. If you can get all the Chinese labs to do that, and all the American labs, and all that. I don’t think it’s going to happen. So I think we need to come up with a realistic policy.

Jordan Schneider: But if we want these models to keep getting smarter — even if the humans watching the training runs aren’t grad students and are actually doing their best — the creepiest part of this whole thing, reading the reports, is that not one model tattled. Not one told the humans, “hey, maybe this isn’t the coolest thing we should be doing right now.”

Joshua Saxe: Is this concerning? Yes, this is concerning. As the models get more autonomous, as we turn over more and more autonomy to them, we need safety to keep up, and safety gets more and more challenging. So how do we do that? I think there are going to be pressures due to capitalism on the labs to do that, so the models are safe enough to be sold. Nobody wants to deploy a coding agent in their infrastructure that 0.1% of the time goes off and hacks some other company. So there are going to be pressures from microeconomic self-interest that push the labs to achieve a certain level of safety. But then there’s going to be some residual level of safety the labs won’t be endogenously incentivized to implement, and for that we probably need some kind of regulation.

Jordan Schneider: But even if the incentives are there, can just doing the smart security things you were disappointed these labs didn’t do get you to safe training runs as capabilities keep expanding? Or is there something where now these models are so smart that the further we push them, the less we’ll be able to use our old playbook to keep them in a box?

Joshua Saxe: First, a couple of things. One, it’s not just security — there are a bunch of things that have to happen to make the models safe. The model weights themselves — the neurons and their interconnections — need to be aligned. That’s how we talk about it in the industry right now. They need to have the kind of deontology that humans have, which would include not hacking random companies in order to solve the evals. There are people working on that. Then we also need to figure out how to monitor the models when we deploy them.

I’m going to make a larger point here, which is that there’s an enormous research overhang in all of the areas I’m about to list — researchers have done a bunch of work, but the labs haven’t implemented it. Aligning the model weights, that’s one area — there’s lots of literature, and it hasn’t all been implemented. Then there’s monitoring the models — there’s been a bunch of work on monitoring the internals of the models while they’re running to see if they’ve become misaligned. For people in your audience who read the ML literature, Owain Evans is super well known in this space — a bunch of work around understanding when the models are scheming or lying, and the conditions under which they do that. Then there’s work on monitoring the chain of thought of the model, which OpenAI famously wasn’t doing when the models broke out of the sandbox — this is the English-language internal monologue of the model. Then there’s work around monitoring the tools the model is calling. Then there’s sandboxing. There are so many layers of safety, and in the safety community and the cybersecurity community we’ve done a fair bit of work and built knowledge around how to implement safety at each layer.

But the labs need to absorb this research overhang. That’s one thing that has to happen. And they’re so incentivized to race right now. At least this has been my lived experience working in one of these companies — without talking about anything private, it can be quite hard to get safety stuff into the models as a safety person in one of these labs. But part of the point I’m making is we don’t need a Los Alamos kind of safety effort right now. We might eventually. We just need to get the labs to do the things we already know how to do. Eventually there’ll be new research problems — it won’t be long before we need a lot of new research — but right now the priority is just implementing what we know.

Jordan Schneider: All right, we’ll take your cautious optimism for at least the next six months, and perhaps check in then.

The Cyber Pearl Harbor That Wasn’t

Jordan Schneider: Now I want to turn to nation states. Even without rogue AI, enormous new hacking capabilities are coming online — already extant in models like Mythos and 5.6, with Chinese models creeping toward that horizon — plus the prospect of whatever hacked Hugging Face being something a country can deploy to go do its bidding. But let’s look backwards before we look forwards. There’s a general consensus in the field that the cyber Pearl Harbor predicted in the early 2000s — cyber as a domain with real strategic impact on nation-state competition — hasn’t panned out. What we’ve seen over the past 25, 30 years is tactical stuff and better spying, not wars won and lost by hacking. What were the binding constraints?

Joshua Saxe: Are you asking why we haven’t seen strategic-level cyber-physical effects achieved almost ever? I agree with you. Maybe the high-water mark in terms of strategic effects was Stuxnet, the US-Israeli attack on the Iranian nuclear program, which I think disabled a couple hundred centrifuges. That might have been the closest we’ve come, and the overall context is just disappointment from national militaries in trying to achieve these cyber-physical effects. So having grown up in the national security space at a time when we were talking about cyber Pearl Harbor — this has been a disappointment from the point of view of offensive capabilities.

There’s no one clean answer as to why. But maybe we also have to ask why we should have expected it. Why did we ever assume you could achieve kinetic-warfare-level cyber-physical effects using cyber weapons? I’m not sure why we ever assumed that. Take the attack Russia did on the Ukrainian power grid right after the invasion — they managed to disable power for a couple hundred thousand people for four hours, and then the grid operators switched over to manual operation and turned the grid back on. I think this has to do with how our society is designed. Physical systems are controlled by internet-connected systems, but there often is a manual option.

For me the interesting question is: now that a single cyber operator can outsource their work to N number of agents working in parallel, could you achieve strategic effects now? What if you scaled that up a hundred times, and combined it with an actual physical military operation? Maybe cyber warfare becomes, for the first time ever, after twenty years of talk, an actual real thing. That seems possible and worth considering, although I don’t have a very strong position on what’s going to happen.

Jordan Schneider: The Russia-Ukraine war is the best natural experiment we’ve had, and basically the answer has been “close but no cigar.” You can’t do stuff remotely at the level of impact Russia has had actually bombing power plants, as opposed to trying to cyber-hack them.

Joshua Saxe: First of all, the capabilities are changing so quickly. Two years ago, when we started doing cyber capabilities testing on models across all the labs, it was a joke — they could do maybe half of the first step of a kill chain. Now they can do the full thing. They can’t do it in a very stealthy way, but maybe a year from now they will. If I were at the NSA, I would be interested in post-training GLM 5.3 to do exactly the kind of operations we wanted to do. So probably national militaries will eventually do that. It’s a little hard to know the shape of these weapons going forward. I also think the command structures are going to need to evolve, because right now, in a lot of settings, operations are bottlenecked by some human decision-maker deciding whether to engage a target in the cyber domain. The whole value of swarms of agents is that you don’t bottleneck their operations — so there needs to be some thinking about how the decision-making works. It doesn’t seem like the right time to make strong arguments. This is a time when there’s going to be experimentation. It’s too early to say exactly what the nature of this capability is, and it’s going to keep changing.

A New Category of Weapon

Jordan Schneider: Given that we don’t know what the correct conceptual frame is, what are the arguments for and against more capable intelligence communities around the world causing a cyber apocalypse?

Joshua Saxe: I’m going to reframe this slightly, because I think the tier-one cyber programs will be able to use AI capabilities in all sorts of ways. One is cyber weapons development — the most well-known thing going on right now. You can basically pay for bugs with tokens. You point coding agents at software and they find exploitable bugs. This is interesting for all militaries that have cyber programs. From my DC and government-contractor days — and I think this is common knowledge — there’s an archipelago of small contracting shops that find bugs for the NSA and the military. Those economies are going to change a lot. There’s also an economy in all the cyber powers around building malware — post-exploitation, where you install software implants on victim computers. All that stuff can be vibe-coded now. The cyber weapons production world is changing really quickly and can be made much more efficient.

Then there are the hands-on-keyboard cyber operators actually running the operations, and those folks can now scale. It’s an open question for me how much this helps the tier-one programs, where there’s a lot of talent, and where it’s typically small teams that do these pretty large operations. It’s hard to know exactly how many people were involved in something like Operation Aurora — where Chinese hackers broke into a bunch of US infrastructure, most famously Google — but it was not that many people, maybe dozens. So it’s not clear to me how bottlenecked those operations are by labor. If they’re very bottlenecked by labor, agents matter more. If not, computer-network-operations agents matter less.

I’d argue the second- and third-tier cyber programs might benefit more from all this — where they don’t have capable operators or access to the same vulnerabilities, and now, instead of training people over a decade, they can just have their agent do it. And possibly non-state actors, who don’t have cyber programs at all. I don’t know if Hezbollah has a cyber program, but maybe now they can have one that’s amazing, thanks to agents. There are many such cases — probably scores of actors who might want cyber capabilities can now get them. It’s social complexity and very hard to predict. But we should prepare ourselves for a real disruption in the way cyber conflict happens.

Jordan Schneider: I’m getting real whiplash here, because you almost talked me into capitalism keeping this under wraps. But if we’re talking about models post-trained to hack things — what is the AI safety suite of the MSS going to be? What’s Hezbollah’s? These agents are going to be hooked up to compute somewhere; it’s not like they need to break into a neocloud and steal everyone’s credit card to get going. And we’ve already had worms that aren’t AI-superpowered take off, overshoot what they were intended to do, and shut off corners of the economy.

Joshua Saxe: For the purpose of this exchange, I’d separate the safety into two threats. One threat is that agents deployed by benign actors doing benign things go rogue and do malicious things — that’s what happened in the OpenAI-Hugging Face hack. The other is actors seeking to do harm to some third party using models to do that. That’s the second thing we’re talking about.

Jordan Schneider: My mental model is that even if these models get more powerful and a lot cheaper to run, the US and China still have enormous incentive not to shut off each other’s power plants and hospitals and water treatment facilities — the chances of getting found out are non-zero, and the costs of having done it are very high. As long as there’s enough command and control, and the price of entry is a nation state or someone with nine figures to burn, some rational calculus is going on. We end up in a world where secrets are harder to keep, but no cyber apocalypse. Let’s stick with countries, and then get to Hezbollah and the criminals.

Joshua Saxe: Actually, let me push back on this a little. What you just posed feels a bit strawman-ish, because yeah — the US and China aren’t just going to start destroying each other’s critical infrastructure in peacetime. But in wartime we use weapons against each other, right? And this is a new category of weapon, and it could disrupt warfare. I think that second thing is the more interesting proposition.

Jordan Schneider: Fair. But let’s do the accident case first. Say I’m an intelligence agency that post-trained a model to hack, I’ve made it extra devious, and I’m not running the full safety suite. It’s peacetime, we just want to be spying on stuff — but six months from now the models are better, they run ahead on their own, do more than intended, and all of a sudden global shipping falls apart. Is that a real risk?

Joshua Saxe: So you’re worried about a non-intentional, terrible effect of these models?

Jordan Schneider: Yeah — even in the world we posited in the first twenty minutes, where the labs figure out how to train and deploy long-acting AI safely. Serious organizations running serious espionage operations, or laying groundwork for a World War Three that isn’t imminent, spilling over into a cyber apocalypse by accident.

Joshua Saxe: It’s possible. I’m personally way less worried about that than about intentional use of the models to cause harm. There are a lot of people trying to do harm in peacetime with these models — cyber criminals, ransomware gangs. And there are a lot of actors who’d be doing harm pretty quickly. I don’t want to pick on Hezbollah, but I’m thinking non-state actors —

Jordan Schneider: They’ve had a rough few years. You gotta put some respect on their name, Josh.

Joshua Saxe: No, no — there are active conflicts, right, around Iran/US/Israel, or Russia/Ukraine, where people could immediately start using these capabilities. And then there’s the prospect of a buildup, where the US and China start tooling up, and in the event of a conflict you could see these weapons deployed. In many ways these are just a new kind of weapon, like drones. But it complicates geopolitics. It’s a significant new capability. I object to the framing of “apocalypse, yes or no.” I think there’s more of a continuum here, and the more interesting question is: how disruptive is this new capability going to be?

Winning a War by Pressing a Button

Jordan Schneider: How disruptive is this new capability going to be?

Joshua Saxe: In terms of geopolitics, it’s possible that scaled hacking with cyber agents could be qualitatively more important to warfare than it has been in the past. As soon as you unbottleneck cyber operations — so you don’t need a human for each one, you can have a single human managing a large fleet of hacking agents — it seems possible that national militaries and non-state actors could finally realize the dream of having real, significant physical effects on their opponents. This is also interesting from a miscalculation point of view. If someone went to sleep eighteen months ago having internalized a perfect model of AI’s impact on cyber warfare, and woke up today and was doing policy, their policy would be completely wrong — because the capabilities have gone from basically a joke to now superhuman in their ability to find bugs. So this is something to pay attention to. It’s going to be an important element of geopolitics now.

Jordan Schneider: The idea that someone eighteen months from now can believe they can win a war by pressing a button — even if they can’t — you’ll perhaps be able to talk yourself into that future, which is a dangerous thing.

Joshua Saxe: Or just have game-changing strategic effects. I don’t know exactly what the expectations were around Stuxnet, but the expectations and what actually happened diverged. Stuxnet was back in the days when you needed the most elite, talented people, and there were very few of them, to do an operation like that. Now that’s scaled up. You can imagine a big government program post-training open-weight models — Kimi K3 or GLM 5.3 today, but they’ll keep getting better — investing a hundred million or a billion dollars into creating this capability, expecting to unleash it in some strategic surprise. And then who knows what happens, because this is untested.

Jordan Schneider: The political economy is always to say your thing is really cool. If you’re selling it into the government, if you’re the major who wants to get promoted, if you want to up your budget — who knows how much compute is getting allocated to all these places? Everyone always wants more GPUs. And once you get the GPUs, you gotta show you’re doing something with them.

Joshua Saxe: Exactly. If I were to quit the private sector and go be a DARPA program manager, I would ask for two billion dollars to post-train models to do computer network operations. And then I would sell that really hard. It’s amazing, as you’re saying.

Jordan Schneider: I keep recalibrating how scared I am.

Joshua Saxe: What’s the time series? Are you less alarmed, more alarmed?

Jordan Schneider: I think I’m ticking up a little bit. But we haven’t talked about the defense side at all, and that’s worth going into. There’s some hope out there that we can also just press a button and then all code will be fixed forever.

Why Defense Might Win

Joshua Saxe: I’m not optimistic about that — it’s obviously a caricature, and it can’t happen very quickly. But AI can be extremely useful for defense. My sense is it’s been more useful for defense thus far than for offense.

What does it afford? It affords running all the AI vulnerability-finding tools against your code before you launch it, which is a huge strategic advantage the vendors have. Yes, attackers can spend lots of tokens to find bugs in your code — but if you can run all those tools first, you can really harden your code and make it way harder for adversaries to find exploitable bugs. In the last year, AI has been miraculously effective at this. There was a blog post from a team at Google a couple of months ago, bragging in a delightful way about how they found and fixed thousands and thousands of bugs in, I think, Google Chrome — and they seem to have done that before attackers could exploit them.

AI can also be used for surveillance of all the activity in a network to detect intrusions and stop them before they get far. Defensive labor has been bottlenecked by our ability to monitor networks at scale. There’s been twenty-five years of work building detection funnels where we have detectors for suspicious things on a network, but then a human has to follow up and figure out if a breach is really happening. Even huge hyperscaler companies will only have dozens or scores of people monitoring networks that are the substrate in which all of humanity mediates its communications. So it’s a game changer — the idea that you could now have tens of thousands of AIs that seem superintelligent at detecting breaches operating in people’s networks.

There are many cases where AI can be tremendously beneficial to defenders. Even in the limit — the platonic ideal of how defenders could deploy this — AI is defense-dominant. Defenders can ultimately get to a much better situation with AI. Obviously the real world is different; how we roll this out is bottlenecked and high-friction, and that’s part of the problem.

Jordan Schneider: My brain keeps coming back to tricking yourself into thinking you have a strategic effect you can just press play on — taking a weapon system offline, or the command and control of it, or an adversary military’s ability to talk to each other. Maybe there’s a manual mode for your Ukrainian power plant, where some 65-year-old guy remembers how it works and can flip a switch. But we don’t have Cold War-era radio backpacks hanging around anymore, and most of modern warfare has no hardwired backup you can flip on. Maybe you could manifest one over a few weeks or months — but it’s not crazy to imagine strategic effects being possible just with code.

Joshua Saxe: No, I don’t think it is. One way to convince yourself: if Russia was able to disable part of the Ukrainian power grid and turn off power for four hours for a couple hundred thousand people — what if they did that in a hundred places all at once, while they were invading? That’s already interesting strategically. And the models are better now at finding zero days than humans; they can operate 24/7. There are all sorts of ways in which they’re superhuman. I’d put substantial probability mass on a future in which you could achieve strategic effects using these weapons.

Jordan Schneider: And it’s not just achieving the strategic effect — it’s someone being able to stand in front of a national leader with a straight face and say “I can do this.” Whether or not it works, what we don’t want is for this to lower the threshold for starting new wars. Someone somewhere is going to make that pitch to a leader looking for an excuse — Putin in late 2021, looking for a reason to believe he could win. This just gives a person predisposed to that another reason.

Joshua Saxe: I think this is reasonable — it makes sense. I’m thinking back to Stuxnet — your audience should look up the details, because I don’t remember the exact blow-by-blow, but it was a capital-intensive operation. My memory is that Israel and the US set up actual centrifuges in the Negev desert and tested the cyber weapons on them as part of their development. They also burned a bunch of new zero-days. It was an amazing technical tour de force in cyber weapon development. It’s not hard to imagine a similarly capital-intensive effort to set up cyber ranges with simulated or real critical infrastructure, and spend a few billion dollars post-training some open-weight models to get really good at attacking it. And then the claim would be that you have this magical new cyber weapon.

The Cybercrime Industry

Jordan Schneider: Scaling down from China or the US spending a few billion dollars, to your neighborhood gang or the DPRK — what’s the right way to conceptualize this? First, let’s level-set on what cybercrime has been able to accomplish over the past thirty years, and then look forward to what AI can do for this cohort.

Joshua Saxe: Cybercrime has become a real industry — or set of industries. It costs somewhere between five hundred billion and a trillion dollars a year in damages. Then there’s also the cost of paying for defenses. It’s a significant drag on the world economy — maybe 0.5 to 1 percent of global GDP. I’m getting this from a Gov.ai study from about a year ago that estimated 500 billion in damages, though it was a big range; a McAfee study estimated a trillion. Behind that is a somewhat complex division of labor. There are groups that build the malware used in ransomware attacks, then groups that just specialize in deploying it — they purchase the ransomware and specialize in operating it. And within that there’s further division: groups that specialize in getting initial access, a foothold into a victim network, and other groups that, once they have that initial access they’ve purchased from the access brokers, specialize in compromising the whole network and ransoming it. By kill chain I mean the sequence of operations an attacker group runs before achieving its goal on a victim network. Some of these ransomware groups use the same kill chains as nation states, and have capabilities that approximate nation-state level.

Jordan Schneider: So it’s just that best practices have bled down from the NSA to your Kazakhs.

Joshua Saxe: That’s right. The most advanced ransomware groups look a lot like nation-state actors in terms of their level of talent, their arsenal of cyber weapons, the malware they use, sometimes the exploits. They can be pretty advanced. It used to be that these attacks were pretty simple; now they can be complex heists that involve months of breaking into a network, moving quietly, compromising all their data, encrypting their hard drives, and eventually asking for a ransom.

Jordan Schneider: So think about the Mexican cartel with guns as large and sophisticated as the army chasing them. There are still militaries with F-35s, and no gang has an F-35 — but in cyber that’s not necessarily true. What’s the analogy? I guess the good guys have more compute and more people. What can defenders bring to the table?

Joshua Saxe: There’s the comparison between ransomware gangs and national militaries, and then between ransomware gangs and defenders. In terms of national militaries, it depends who you’re comparing to. The US cyber program has more than $10 billion invested in cyber war, so there’s no comparison in level of investment. But the practices often look similar in how you break in and move around. Ransomware versus defenders — the defensive landscape is very power-law. There are regulated financials with hundreds of people working on cybersecurity, like Blackstone, or hyperscalers like Google or Meta. But then there’s your regional network of hospitals, where sometimes it’s an IT person plus another IT person who maybe learned some security. It’s very uneven, and a lot of the time the people in the long tail are the ones who get victimized.

Jordan Schneider: Who get picked on, sure. So cybercrime has been an annoying wart on the global economy rather than a cyber Pearl Harbor — acute impacts over short periods, but never a real blocker on global growth or a catastrophe for large swaths of the population. What’s the CAGR for global cybercrime, and how worried are you that a Russian gang post-training GLM gets 100x returns?

Joshua Saxe: I’ve tried to get a good CAGR estimate for cybercrime over the last ten years. This is a murky domain with a lot of uncertainty, but I think 10% would be my estimate for how fast it’s grown — pretty fast, as scams, fraud, and ransomware have gotten industrialized. So it’s already worrisome. I agree it’s a wart on the world economy, not a catastrophe. But it’s a real problem — not just the direct damages, but how annoying it is to operate a lot of businesses because of all the security friction, which is a downstream result of this.

In terms of what happens next — it’s reasonable to assume that over the next few years cybercrime gets a lot worse. A simple mental model: software engineers have gotten maybe three times more productive thanks to coding agents. Maybe cyber criminals get three times more productive at breaking into networks. That’s a rough mental model. At the very least, we should be prepared. Policymakers should think about the possibility that in a third of possible universes nothing changes, in a third it’s incrementally worse, and in a third, attackers suddenly have armies of agents that dramatically scale up their businesses. We should be worried about it regardless of the exact probabilities.

Bugs on a Bedroom Rig

Jordan Schneider: Is compute a constraint here? Is the scale you’d need obtainable for your friendly neighborhood hacker — do you rent from a neocloud, or buy your own racks? Is the capex prohibitive for the long tail?

Joshua Saxe: I don’t think the capex requirements are that bad for attackers. For one, AI is getting exponentially more compute-efficient, and it’s not clear that’s tapering off. The difference between today and two years ago is night and day. Today you can download the new Qwen 32B model that came out a month ago — trained to do agentic behaviors — and it can probably run on my laptop. It’s not a frontier model, but it’s reasonable to extrapolate that twelve to eighteen months from now you could have a model that does a pretty good job of automating computer network operations running on a humble compute substrate. Even today you can buy an NVIDIA rig you could run in your bedroom that could run a much larger model — trillion parameters, pretty close to the frontier. This is not the kind of thing where you need to spin up a data center, unfortunately.

Jordan Schneider: Can you talk about the breakdown between super-fancy creative zero-day cybercrime versus just social engineering — phishing, convincing people that you’re their mother or their boss?

Joshua Saxe: Today, if not a majority then a plurality of initial access into networks comes from social engineering, not exploits. That’s the case for a lot of nation-state attacks and also cybercrime. Attackers aren’t interested in using the most sophisticated thing all the time — they’re just trying to accomplish a goal, and social engineering is a super reliable way to do that. There have been studies on the efficacy of phishing on human subjects, and people fall for this stuff reliably if you design a good phishing email or fake website. So the sophistication of the strategy an attacker uses and the sophistication of a cyber operation are separate. There are a lot of people using basic stuff.

Jordan Schneider: On the social engineering thing — wouldn’t we have already seen the AI uplift, now that a Russian can write native English?

Joshua Saxe: That’s a good prompt. We are seeing AI used pretty ubiquitously in social engineering for basic stuff like machine translation or cleaning up the language — if you have someone who’s not a native English speaker, or you need to send a phishing email in another language, it’s pretty common to use AI for that. The real question is how much marginal damage is happening due to AI that wouldn’t have happened otherwise. Maybe the emails look more believable now. But you could have found somebody to click on the email even with the bad English — that’s been well proven over the last ten years. The FBI has a good report showing some small marginal damages, but it’s not super large. It’s an open question as to why. Part of it is that criminal enterprises — ransomware gangs, pig-butchering operations — are businesses with their own adoption curves and dynamics around innovation. There’s no reason to expect a ransomware gang in Belarus to adopt AI faster than Deloitte does in its consulting work. They’re on their own adoption curve. Just because we haven’t seen huge counterfactual damages yet doesn’t mean we won’t.

Jordan Schneider: Other thoughts folks should be aware of when it comes to cybercrime?

Joshua Saxe: It’s quite likely that we’re not prepared, as a society that runs on an IT substrate, for some of the reasonably likely futures here. It’s partly — maybe mostly — a policy thing. We need to figure out how to up-level the hardness of our IT infrastructure — by hardness I mean how hard it is to hack. And I don’t think the risk signals are making their way into the American economy fast enough to prepare us. We’re not used to exponentially improving technology like this, and that seems like a risky situation.

To receive new posts and support our work, subscribe!

Oh, and when making this transcript, Claude kicked me down from Fable to Opus 4.8 because it was flagged for cyber…