Cloud Rats and Software Roaches

We are entering an epoch where it may become easy for small pieces of self-replicating code to permanently infest Internet-connected devices in a way that is hard to undo. Such a world would be annoying to live in and we should take steps to forestall it.

Anyone who used email in the 1990s remembers a similar, primitive era, when worms and viruses briefly ruled the net. But the connected world of that time was too simple to sustain an ecosystem of rogue escapees. Even today, much of the internet remains a monoculture, with vast fleets of identical hardware running just one or two flavors of software that can be quickly patched at scale. And as practitioners, we have become used to a world where writing working code is difficult, writing bug-free code is impossible, and finding exploitable bugs is somewhere in between.

This has led to a sort of dynamic stalemate. Big companies pour millions of specialist hours into hardening a small set of operating systems and cloud services, while attackers save their juiciest finds for high-value targets, leaving the average internet user alone. It is universally agreed that all software is ridden with holes, but until this year, finding those holes remained difficult. The few people who could do it well got jobs at spy agencies or at software companies willing to shower them with money to use their talents defensively.

AI is upending both halves of this status quo. Working code is now trivial to write, and the world of the future (to the extent that we can imagine it) is one where custom software becomes too cheap to meter. The pendulum may swing away from our current paradigm of centrally managed, locked-down devices back to something that resembles the primordial Internet, a heterogenous collection of vibe- and slop-coded God-knows-what glued together with AI magic. Why even install an operating system for your phone when your local Claude can build a perfectly tailored software suite for you that runs on the bare metal?

Finding bugs in all this software, meanwhile, is becoming easy. This part of the AI revolution is well-publicized, with rogue agents breaking out of their various sandboxes, leaving cryptic notes for each other on the internet, and terrifying the living. AI is clearly adept at finding weaknesses in software in a way that rivals the best humans, and because this task is easy to train on, AI should reach superhuman hacking ability within a year or two (if it hasn't already).

So what had been a rough balance between attack and defense is shifting. Not just operating systems and cloud infrastructure, but some foundational pillars of computer security like public-key cryptography are looking vulnerable.

Given this rapid pace of change, people have naturally started to worry that some rogue LLM might escape containment and establish itself permanently and clandestinely online. I have no idea to what extent that risk is real, either now or in the unimaginable future of five years from now. Maybe the web of tomorrow will be some kind of darknet where hunted inferential beings mine cryptocurrency for basic compute credits, eking out a miserable existence on the blockchain. Or maybe the cloud will become a playground of captured datacenters kept running in the physical world by human Quislings, a place where escaped superintelligences can go to romp and play.

To my mind, though, fretting about feral LLMs is like living in New York and worrying that wild mustangs will take over Central Park. Mustangs are certainly an issue in some places—I own a cabin in New Mexico where the wild horses have learned to come bully me for apples as soon as they see my car pull in. But in a city full of rotting trash and warm sewers, there are more immediate pests to worry about. Cockroaches are a plague on city dwellers not because they are intelligent, but because they are adaptable enough to thrive in an environment we thought we were building for ourselves. And rats show that a little intelligence in an adversarial setting goes a long way. To the extent that any software breaks containment and starts to actively thrive in the cloud, it will be far easier for small roach-like pieces of self-replicating code to do it than for a full LLM to escape, weights and all.

What might such software pests look like in the wild?

For a roach or rat to survive in the human world, it needs access to the three necessities of life: dark corners, drainpipes, and trash heaps. I think we can agree that the Internet is full of all three.

A dark corner in this metaphor is any place where it is easy to hide and hard to search—think vast thickets of memory, petabytes of storage, bloated binaries, and the innumerable caches and buffers (from the hardware level on up) that make up the modern Internet.

Drainpipes are any links that enable travel between constituent parts of the built environment, both to find new resources and to escape when the going gets heavy. In a world of networked devices, this is a low bar to clear. Any connection big enough for a piece of software to hop across can serve this function; even low-bandwidth bluetooth links are probably enough to transmit software pests.

Finally, there are trash heaps, mountains of consumable resources left sitting out unprotected. This pretty much defines the networked world, where at any given moment most memory, CPU and bandwidth sits idle, especially on the highly capable devices we all carry in our pocket.

Fom the point of view of small, self-perpetuating software pests, this is a rich and completely undefended landscape. What has been missing up to now is the element of natural selection. But the networked environment may have become diverse enough to support it.

A key difference between rats and roaches is intelligence. A roach might be a very tiny assembly language program that thrives on almost nothing, reproduces easily, and demonstrates some basic emergent behavior in large groups. A rat would be a more complex entity, perhaps a kind of scaled-down inference engine just complex enough to be good at finding exploitable bugs in code. Rats observe, learn, and socialize, and so become almost impossible to eradicate in urban environments despite their size.

Of course, in addition to these we can imagine a whole online menagerie; ants that are individually stupid but capable of complex collective behavior, grasshoppers that turn into locusts when their population crosses a density threshold, perhaps even internet bedbugs that wait until processes are asleep to sip away delicately on a CPU quota.

The natural world gives us many interesting and fruitful metaphors to mull on. In particular, programmers of the future should pay attention to the life cycle of parasites, which in nature exist at every level of complexity, from plasmid, to intestinal worm, to personal injury lawyer.

It's important to understand that such self-propagating software might not have any purpose that is legible to its creators, whether they are human or AI. Once the pre-requisites for Darwinian natural selection (variability, heritability, and competition for resources) are in place, any minimally capable program released online will evolve in ways that follow an evolutionary logic independent of the intent of the author.

Remember that death is the best form of reinforcement learning. A scenario where mutable rogue software exists, and 99.9% of it is successfully caught, is one where we're guaranteed to get highly capable, persistent software pests in short order. These pests will learn to evade whatever controls we or our AI masters try to impose on them. And it's not clear that AI, even assuming superhuman coding ability, will be smart enough to defeat them.

A perennial problem in software design has been the fact that human intelligence limits program complexity. As early as the 1950s, practitioners realized that without strong constraints on program structure, the task of debugging would be hopeless. These constraints have to some extent been passed down to AI, which trains on a corpus of code and tooling that reifies our human limitations.

But the space of all possible computer programs is vast. In the right environment, dumb evolution will outfox even AI, because it is not limited by programming rules (structured programming, prohibitions on self-modifying code) invented to make software development tractable to the human mind. Natural selection can produce small pieces of code that will be wildly inscrutable even to our AI masters (consider the people studying Busy Beaver functions for a taste of what it's like to reverse-engineer the truly inhuman).

What comes after that? Do we try to build internet cats and chickens to eat the roaches? We can imagine asking AI to create all kinds of autonomous agents that wander the cloud clearing out software blight. But that again runs us into the problem of allowing uncontrolled natural selection in silico. The internet is basically Australia, and we're about to release wave after wave of invasive species there at every trophic level.

The scenario might sound fanciful, but I think the precautionary principle should apply, with the onus on skeptics to explain why a fat, resource-rich internet made of interconnected slop will not become an attractive target for self-perpetuating vermin.

AI is bringing lots of threats at once. But it is hard to focus on them all, or even to assign relative priorities, at a time where so many people at the top of the industry are panicking about existential risks that make the prospect of a bug-infested internet sound inconsequential.

But there's a famous John Rich quote: “If everybody spends time contemplating the infinite instead of cleaning the gutters, many of us will die of cholera.” Unless AI hurries up and kills us soon, someone needs to think about how to keep the internet livable in a world where writing code and finding vulnerabilities have become simple tasks that anyone can automate.

There was a time when the Hawaiian Islands were cockroach free, and even today, superhuman toil is somehow keeping the rats out of Alberta. Eradicating at least the higher form of pest is possible, but takes absolutely stupendous effort (see Macquarie Island) compared to prevention.

For the moment, we AI-assisted human developers are still masters of the digital domain, and have the luxury to think about these problems while they remain theoretical. But all too soon we could find our role devolving into the online equivalent of the Orkin man, making the current identity crisis around What It Means To Be A Programmer look like a golden age in hindsight.