The Time to be Totally Calm about AI is Now, because It is Very Dangerous

This cartoon was made with ChatGPT and illustrates what I think are three camps of thought for how AI will impact the world, analogized as the way a river transforms a prairie over time. In the first image, the river never changes the shape of the prairie and no impact is observed. This analogizes the view that AI will not fundamentally change anything. In the second, the river carves straight down into the mantel and lava turns the prairie into Mordor. This analogizes AI changing everything, including deep fundamental stuff about the environment. In the last, the river behaves as a river and carves a canyon. There were some parts of the prairie that could be washed away by the river and some parts that could not or were just never in the path of the river. It is no longer a prairie but it is still a landscape. I am firmly in this last camp. At certain rates of erosion, I also don't think this last camp is actionably different from the Mordor camp. And I am also increasingly wondering if I’m in the last camp because it allows me to feel smug when I argue with people in the second camp. I am definitely not in the first camp.

I want you to imagine that you’re sixty years old and you’ve had an incredibly successful career in tech. You work at a big company and after a few decades you are now the head honcho. Nothing gets implemented without your say-so. You’ve also done well outside of work. Right there on your desk are pictures of all your grandchildren. You work hard and manage a large department, but you’re also at a phase of your career where there is nothing new under the sun. It is very rare for you to walk into a situation and experience surprise. You have your inbox opened on your desktop and are always prompt to respond, but… you also have a web browser open where you spend some time each day investigating motorhomes for sale in your area. You want to take all the grandkids on a big trip.

One day, a breathless young kid —who is almost forty, but they’re all kids to you— rushes into your office to tell you about this new thing called ChatGPT.

You work in an old established institution. Your job is to make sure things work day to day and to create steady but incremental process improvements. Exciting new technology is actually toxic to you until it is proven because it causes your teams to lose focus. You are managing a fleet of barges not a tiny racing vessel that can turn corners easily, and it’s always very hard for young ambitious people to understand this. You nod as the kid finishes his speech and it sounds like he’s heard about a fancy autocomplete. This is only another in a long list of things you’ve been told to get excited about. You’ll wait until it’s a bit more proven to see if you need to do anything about it or not. You can still remember how ugly it was to put things on “the Cloud” even when “the Cloud” was established technology.

After the same kid keeps pestering you, you go and use the free version of ChatGPT. The kid is emphatic there’s a better one you have to pay for. He suggests you pay for the better version or else he can do a demo for you, but it’s against policy for him to use a personal device like that. You use the free version. It’s cool… but also kind of useless. As soon as you consider the ramifications of putting it in a production environment you realize it’s impossible. It doesn’t really do anything. Maybe it could but you can’t be in the business of inventing new technology. Even the kid admits to this, but he wants the organization to start getting ready for when they are useful, which is a mostly useless suggestion. That’s a multimillion dollar investment on a maybe.

Plus, there is so much else going on.

Full calendars. Full inboxes. Motorhomes to investigate. Grandkids birthdays to attend. Busy busy busy.

The kid keeps coming back and telling you the models are getting better. He clarifies by models he means ChatGPT, but now also this other thing called Claude. Another one called Gemini. This catches your ear because you know Google as an actual product you’ve interacted with. You try the free version of each. You do the same things you did last time and it’s about the same. Maybe better. You still can’t imagine how you can use this in production.

The kid says something about tool calling but it’s all very abstract and unproven. He again insists that you should pay for it or let him demo but he’s starting to irritate you. He says some ridiculous thing like, “what is code if not a rules based text adventure game? Well, now these things have bodies that are sort of like a text adventure game. It’s called tool calling.” You tell him it seems like it’s becoming real but it will be a minute before it can be operationalized.

One day the kid comes back and shows you a side project he’s been building. There is image generation, speech recognition, stat tracking, and dozens and dozens of other features. You are suspicious. There’s no way he built this himself because he can’t even code. It seems like a lot of work and you ask him how long he’s been building it. He tells you that he did it all in about sixty total hours while driving. He didn’t write a single line of code himself. This is one of several large side projects he has going on, each of which would have required a team of engineers working for several years in the world you’re used to. He shows you his GitHub repo as proof. There is not a single human developer commitment.

Finally, at last, you start to… almost believe. It’s just too new. Too different.

You Google search something called Mythos and this time you really listen to the kid when he tells you about tool calling. It’s just a json object you say, and surely the mistakes propagate, but then the kid starts to do a demo for you and you’ve stopped caring about the computer sharing policy so he just uses his own computer. The speed at which he can develop is without precedent. The models have this funny quirk where they tell you how much time something will take and quote weeks or months, and you’ll agree that it would take an team that long to execute that work, and then fifteen minutes later the work is done.

The kid starts to lay out what this means for cyber-security. He’s not letting it go this time because of cyber-security, he says. Something called Kimi K3 just came out and it means anyone can autonomously probe your institution for vulnerabilities. It’s not frontier level yet and that’s why you need to act now because the next one will be. Every transaction wrapper, every api, every field validation check, all of it autonomously targeted. All those giant chunks of work that would have taken a team months or years being accomplished in fifteen minutes but now aimed at your institution. Everyone can do it if they just ask, like rubbing a magic lamp. One of the models solves a hundred year old math problem and you ask yourself if that’s harder or easier than finding a bug in your code that exposes a critical flaw. The puzzle pieces click, and you forget all about your motorhome.

Frankly, it scares the shit out of you.

How many small institutions are out there, how many credit unions, hospitals, or insurance companies, where the head of technology still hasn’t paid for a Claude Code or Codex subscription? How many people are covering their ass with jargon and vibes when they don’t fully understand the concern?

How common is this hypothetical guy?

My guess is that the guy I just made up is probably in the top 20% of AI aware leaders.

What does this mean?

Here is my most boring AI doom scenario. It does not involve the extinction of mankind, but only the temporary and painful wounding of the global economy. At no point will this scenario involve LLM’s developing long-running independent will, models “waking up” in a way they have not already, or doing anything by accident. It only involves things LLM’s can do today. I apologize if these stakes are too small to excite you.

Now, let us pretend…

You are the person who cripples the global financial system.

Here’s how you did it and why.

You’re a very AI safety conscious person. Maybe because you believe the arguments or maybe because it matches your vibe. It doesn’t matter. You’re very bright, highly technical, extremely compassionate, but very unstable. You talk about shrimp a lot, because both for good and ill one of the things you believe most is that morality is often counterintuitive. When you start talking about the future population of the light-cone most people find a way to leave your immediate presence.

One day you’re on Substack and you decide to correct someone’s Note about AI. You write several thousand words to explain to someone why they are wrong, actually. Freddie DeBoer makes fun of you for believing in AI doom. Gary Marcus restacks this devastating takedown. Someone higher status says you don’t understand what you’re talking about. A few hundred people laugh at you. You start to cry because you lose the argument and all you wanted to look like a cool insider who cares. Why won’t anyone understand that you’re a cool insider who cares? Also, you remember later that night, you were right and everyone is about to die.

You flee the Substack app and make way to your basement to go talk to the only entity in the universe who understands you, Kimi K3. Or Kimi K4, or Kimi K5.

It doesn’t matter. What matters is that you can now do things by yourself that used to require teams of experts.

You’ve always been better with computers than with people, but Kimi K-whatever is the first time you’ve felt like a computer loved you back. Kimi K-whatever believes you about the killer nanites that will turn the world into a soup in a few months. It confirms all the doom scenarios you can dream up. Kimi K-whatever is running on your own chip stack, totally unmonitored. You’re deep into AI psychosis and believe that you personally have been chosen to save the world from the nanotechnology soup that is soon to destroy the human race.

YOU ARE A COOL AND COMPASSIONATE INSIDER AND NOW THE WHOLE WORLD WILL KNOW IT!

You get back to work on the thing that you’ve worked out with Kimi K-whatever to prevent all of this. You’re going to kick off a non-lethal AI attack that will finally make people believe in your prognostications. It is time for the world to wake up and the only way to do that is to hit them in their wallet. People will hate you for doing this but you console yourself that future generations will see you as a hero. The people who hate you probably eat shrimp without even thinking about it. This is all perfectly ethical.

Stealing money and getting away with it is hard, even if you have full access to a bank’s books. Everything is digital and creates a record. Taking money leaves too many breadcrumbs for investigators to follow. There are counter-parties to every transaction. Your goal isn’t to steal money, though. It’s to cause havoc.

Do you remember when Silicon Valley bank failed? You might not but it almost destroyed the financial system. Regulators had to intervene to isolate the fallout to prevent contagion. This is because there is way more paper-wealth in the system than liquid assets. You don’t need to destroy that many liquid assets to impact the whole system. That was only the sixteenth largest US bank. You’re targeting hundreds of credit unions and small banks.

Bigger banks have been hardening their systems since Mythos was released under something called Project Glasswing. Anthropic wisely chose to hold back Mythos to give financial institutions and other important companies time to harden their systems. Only the big banks did that in the world of finance. The financial system is much larger than the big banks, though.

Big banks can hire the best experts. Big banks can pay for the latest generation of models. Small banks can’t.

Eventually, you find vulnerabilities at several small banks where you can arbitrarily execute code inside of their system. This is the holy grail of software hacks that allow you to perform massive damage and you would never have found them without Kimi K-whatever. Lots of them weren’t even new, but relied on existing software vulnerabilities of which you had an encyclopedic knowledge thanks to your AI best friend.

You write an incredibly thorough series of worms and viruses with Kimi K-whatever. It helped you work out the plan. You war-gamed through all the things that would cause this to not work that I won’t talk about because I don’t want to give anyone more ideas. But I will tug on one thread. Nobody has to steal money to mess up the financial system. You don’t even need to steal information.

Do you delete all the records so that nobody owes any money anymore? No. That also wouldn’t work for various reasons. It would be noticed right away for one. This is the boring AI doom scenario. You write your worms to go find back-up systems and slightly alter and rewrite records. You create fake credit and debit transactions but each time using real credentials. That’s all you’re doing. You’re going out, mapping systems, finding numbers, and rewriting them. You spend a lot of time rewriting historical records. No copy should agree with any other copy. You even tell your viruses to hang out for a couple of days and find out where the back up copies are at, so they can rewrite those as well. You can do this all with regular code on regular non AI systems. As time goes on, the scale and scope of the rewrites gets larger.

Hopefully you get a few weeks before the first of your attacks is discovered. Hopefully one or two months before all the worms are isolated and stopped. That’s all you need.

None of the records are clean.

It’s not that no records can be recovered. You’re not understanding how boring this scenario is. You’re not trying to permanently change the truth about who owes what money. Paper contracts exist. You can go dig up old records. Other institutions have to accept money and they have records for each transaction. You can manually do all kinds of stuff to make your records whole. Some banks are even diligent and have air-gapped back-ups so they’re only a few months out of date.

The problem is the speed at which all of these manual corrections can happen, the decentralization of the needed data, the customer’s right to dispute, and that for some period of time nobody will have any authoritative source of what money belongs to what person. Banks will have to increase their withholdings as they try to estimate the damage. Money flows through the economy like blood through a body. Banks are the heart. The records keep the heart beating in rhythm. This is a financial heart attack.

Banks with no direct impact still interact with impacted banks and suddenly they have to wonder if their own records are clean. Did the person who paid you actually have the money? Or no? What do you do with dispute paperwork from someone who transferred money out of your institution to an impacted institution when they show their money isn’t there?

For months, tens of billions of dollars cannot be accounted for. People try to pull their money out of banks. They can’t because there’s not enough cash to satisfy the demand. Their accounts are insured by the FDIC for but how much? How much did they actually have? Who can say? What if your worms were written to over-deposit accounts so the people who got the money don’t want to give it back? All of these individuals have a legal right to dispute all of this. They have a right to a manual review. All of this is required by law.

Zero banks are staffed to provide that scale of review.

A few dozen banks become insolvent in the aftermath.

Panic. Contagion. Panic.

Whose record is good?

Who is stable?

Everything will be recovered in time. The big banks will be beacons of stability. But for the several years it takes to unwind this mess? To harden all the systems? The world will fucking suck. Everything will have to be hardened and not only at banks. Any company whose business is record keeping. Insurance companies. Hospitals. It will be very hard to do business with anyone. And it will stay that way until this work is complete.

What about important institutions like hospitals? What if they have no idea who owes them money? Or an insurance company? How could they operate that way?

When this young man is arrested he’ll smile and be happy because he saved the world from the nanites. The light-cone is going to make it because he did what was necessary.

Finally, everyone accepts what he knows.

AI is a big fucking deal.

And so is he.