Computer scientist Cal Newport adds some clarity to the AI debate

As previously mentioned, software veteran Carl Brown has pointed out that we can prevent incidents like the hugging face hack simply by enforcing existing laws.

Here the reliable Cal Newport explains that these incidents all come not from LLMs in general but from the reckless application of one particular application of LLM-powered agents, and that there's no good explanation for the risks these companies have been taking.

When Hubinger says “AI could kill all humans,” he’s not talking about AI in a general sense; he’s really referring to a specific type of AI system that we can call an LLM-powered agent. These systems consist of an old-fashioned computer program that repeatedly does something like the following:

1. Sends a prompt to an LLM describing its goal and current state, then asks for a suggestion about what to do next to move closer to the goal.
2. Blindly executes whatever the LLM describes in its response.
3. Updates its state based on what happens and then loops back to the first step.

LLM-powered agents aren’t necessarily dangerous. Millions of software developers use these systems every day to help write and debug computer code. Though these coding agents make mistakes, no one is worried about them going rogue in any alarming sense. (OpenAI ​recently scanned​ “tens of millions” of traces of coding agent interactions with their LLMs and found zero instances of high-severity incidents.) Hubinger is likely referring to the same sub-class of these agents that was involved in the autonomous hacking attacks over the summer, and which satisfy the following additional properties:

1. They are given access to powerful tools and information specific to computer hacking.
2. The LLM they prompt has had guardrails removed so that it will respond to prompts requesting information about dangerous or illegal behaviors.
3. The agents have minimal (or no) safety checks or constraints on what actions they’ll execute. (Standard coding agents, by contrast, are hard-coded with long lists of commands that they will not execute, even if an LLM suggests it.)
4. The agents are run for very long time periods (sometimes multiple days) without any human supervision. They are programmed to be persistent, meaning that they should never give up but instead continually prompt the LLM for new next steps to try, pushing it to get more creative and brazen in its suggestions.

We can call these long-horizon, dangerously equipped unsupervised LLM-powered agents. As best as we can tell, it’s this very narrow type of unpredictable and potentially hazardous system that companies like Anthropic are rushing recklessly ahead to provide increasingly powerful tools to play with (including, reportedly, the ability to update elements of their own code) and increasing autonomy (achieved, in part, by post-training the LLMs they prompt to suggest more aggressive actions). This information lets us be more specific about recent claims. The concern of the moment is not that AI is inexorably becoming harder to control and scary. It’s instead long-horizon, dangerously equipped unsupervised LLM-powered agents that are making people nervous. There’s an obvious solution here: stop racing to amplify this very specific type of particularly unstable system. This wouldn’t even necessarily entail much financial sacrifice. Most of the things people already like doing with LLMs, or hope LLMs will enable soon, don’t require these haphazard and unpredictable setups. Anthropic and OpenAI could stop working on them today with essentially zero impact on their projected revenue. All of which begs the question: Why have these particular AI companies made these alarming long-horizon LLM-powered agents so central to their efforts? I’m not certain of the answer, but if I were to guess, it probably has a lot to do with the Silicon Valley technological salvation ideology (to borrow a term ​from Adam Becker​) that ​heavily influenced​ key figures like Sam Altman and Dario Amodei, as well as many of their employees. They think these types of agents are their best bet to summon the digital deity of superintelligent AI which, in their futurist eschatology, will either heal the world or destroy it. In some sense, they see themselves as prophets attempting to usher in a new Messianic age. All of this should make you angry.