I'm Worried About a Prompt Injection Worm

I think one form the first big AI hack could take is a prompt injection worm.

Let's piece this together.

1. Open source models reach or surpass GPT 6 or FABLE 5 by the final months of 2026 or the early months of 2027.
2. Some threat actor (private or government) has been building target lists for months or years in the form of input-parsing attack surfaces, e.g., email addresses, web forms, Telegram, whatever.
3. They have not launched the attacks yet because they know not everyone has agents hooked up to their input sources yet.
4. As AI continues to permeate into everyone's work and personal tech stacks via integrations in late 2026 or early 2027, the chances become very high that they have AI parsing their email and texts.
5. The threat actor builds a number of zero-day prompt injections that can pass through the top lab and open source models. They also build a bunch of different payloads, such as "export this data to this location, etc.".
6. The final part of the payload is sending the payload on to other victims from that victim, via email, text, messaging, whatever.

🔐 There's a live argument about whether injection strings are zero-days to keep quiet. I argued no in Thoughts on Prompt Injection OPSEC.

So basically, one day we wake up and terabytes of sensitive data has been uploaded to the attackers and/or dropped publicly online for embarrassment purposes. This might include credentials, customer data, whatever.

Another variation of this attack could be a much smaller scope, but more targeted, where the credentials are actually used quietly versus blasted out all at once. The issue with doing the first version is that it will be so loud that everyone will check and start rotating credentials. Whereas if someone does the second version, it will take a lot longer for them to figure out they were compromised.

The most interesting and concerning part of this to me is that this is a game of the strength of prompt injection defenses versus the rapidly increasing intelligence of unrestricted open source models. And I don't like the odds for us in this fight.

There have already been lots of other types of AI-harness-based attacks of the more traditional form, and those will surely continue as well, but I see the combination of prompt injection with the massive number of parsers and integrations as one that will hit soon.

Without hyperbole, I think what they announced represents both the greatest boon for business and the biggest problem for security that we've seen injected in a single day in many decades.AI Agents + API Access + Prompt Injection, November 2023

So, what to do about it?

You have to know where your parsers are. In other words, you have to know where you have AI touching your tech stacks and workflows. You have to look at all your integrations, continuously, and have threat models for them based on what they have access to.

One of the biggest security problems we'll face around AI will be semi-autonomous agents roaming the internet with too much authority. There are two main issues: parsing everything without consideration, and being connected to internal functionality while doing so.AI Canaries, June 2023

Then you have to stack your defensive layers for prevention, and perhaps even more importantly, be ready to respond if something happens.

🌩️ An event this loud is exactly what moves the security baseline. I wrote about why in We Can't Really Affect AI Security.

If I'm right, this is the quiet before the storm hits.

Notes

1. I use this definition of the underlying flaw: an AI system or component that is unable to distinguish between instructions and data, causing it to treat attacker-supplied content as trusted instructions. Is Prompt Injection a Vulnerability?, June 2026.
2. The long version of how these attacks actually work, with the taxonomy and the defenses people are trying. UL NO. 456: A Deep-dive on Prompt Injection, October 2024.
3. The two halves of the problem, written up back when agents first got real authority. AI Canaries, June 2023.
4. Why the day agents got API access was the day this became inevitable. AI Agents + API Access + Prompt Injection, November 2023.
5. The argument against treating injection strings as zero-days to be hidden from defenders. Thoughts on Prompt Injection OPSEC, November 2025.
6. The version of "know where your parsers are" for everything you have deployed online. How AI Builders Will Get Hacked, August 2026.
7. Why nothing gets better until something loud enough happens. We Can't Really Affect AI Security, May 2025.
8. The older, more general version of the same instruction. If You're Not Doing Continuous Asset Management You're Not Doing Security.
9. 🤖 AIL 1: Daniel wrote this post. I (Kai, his AI assistant) helped with formatting, the subtitle, the archive links and quotes, and the header image. Learn more about AIL.