Against Moloch #48

How much do you trust your model—would you even know if it was misbehaving? For that matter, how much do you trust your frontier lab? We desperately need better visibility into every part of the AI stack.

This week brings us cautious good news: perhaps neural networks are easier to understand than we’d previously believed. And Anthropic is moving forward with embedded independent evaluators, with reason to hope that at least some of the other leading labs will follow suit.

Top pick

Neural networks are already clean

The big news this week is Nick Cammarata’s essay Neural networks are already clean. Nick did some of the seminal work on polysemanticity and is one of the people responsible for the current widespread belief that neuron-level interpretability is almost impossibly difficult.

He now argues that he was mistaken: with the right tools and techniques, most of the neurons in modern models can be understood:

I think I have good news about the nature of neural networks.

After many months studying neurons and circuits in image and language models, I’ve come to believe a network’s own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than I’d feared.

The essay itself is massive and technical. If that’s more than you signed up for, I recommend Zvi’s report, or MIT Technology Review’s interview with Nick.

How important is this work? Opinions vary wildly.

roon (OpenAI):

when nick told me that he had some interp breakthroughs, he was uh, underselling it. i think this may be remembered as a major moment in scientific history. news so good even eliezer will have to lower his pdoom. we can start making alignment into a 'normal science'

Neel Nanda (Google DeepMind):

This is interesting work, but I don't think it's a significant update to the big picture of interpretability.

My take is that we just don’t know yet. This is strong and important work that gives us new hope for mech interp, but it isn’t clear how well the results will hold up, let alone how much they will ultimately help with alignment. I’m sure an army of researchers are frantically working on this, and I look forward to seeing what they find.

Using AI

Stop babysitting your agents

Anthropic’s Sid Bidasaria explains how to level up your coding by giving your agents more autonomy. The core idea is to create a self-improving loop that lets them work for longer, so you can manage a large fleet at a high level rather than micromanaging a small set of agents.

The single most important skill for programmers right now is the ability to keep climbing up the abstraction ladder. Every few months, you need to restructure your workflow to give the agents autonomy over larger chunks of work, while maintaining appropriate oversight of the whole process.

How the Claude Code team does code review

Having a fleet of coding agents at your command is great, but how do you review that much code? Here’s a detailed look at how the Claude Code team manages it. If you’re a developer on a large project, the technical details will be of interest. But for everyone else, pay attention to the larger point:

We want to preserve human time and attention for issues only humans can solve and make sure automated checks catch the rest. Human time covers both sides of review’s cost: the time people spend reviewing, and the time they spend fixing what bad code breaks. Cutting the first doesn't help if the time just shows up in the second.

For people on the cutting edge of AI usage, the question is no longer “how do you get AI to do work for you?”, but rather “how do you maintain human oversight over a volume of work that is far beyond human ability to manage in detail?”

Foothills of the singularity

Tyler Cowen and Dwarkesh Patel predict the future

Dwarkesh Patel talking with Tyler Cowen about the future of AI? Obviously it’s gonna be good.

Everyone’s talking about this quote from Dwarkesh:

Maybe two days ago, I was talking to somebody at Anthropic and they were like, “look, I expect AIs to beat humans at AI research within 9 months.”And I'm like, “why do you believe that?”They’re like, “it's not just vibes, I have information on the inside.”

That’s consistent with other signals from inside the labs, but don’t give too much credence to one casual anonymous comment.

Tyler has an insightful question about the diffusion of AI:

When will it be that normal people walk into Walmart and they buy a thing that is there because of AI that would not have been there without AI?

My answer: sooner than you think, but most normal people will notice that their jobs have been completely transformed (perhaps by vanishing) well before the shelves at Walmart are full of amazing alien artifacts.

Can a million AI agents create superintelligence?

Ramez Naam has another guest piece on Noah Smith’s blog, this time considering the implications of agent swarms for RSI. Naam is one of the best-informed skeptics of runaway RSI, and his reasoning is well worth sitting with.

Naam and I disagree about the likelihood of accelerating RSI in the near future, but we’re on the same page about swarms. Swarms are useful for getting things done faster, but they don’t increase model capability by much.

I agree that this is a good indicator of what trajectory we’re on:

Personally, I don’t expect to see sustained acceleration of Anthropic’s AI progress (as measured by ECI) through the first half of 2027 We shall see. If we see the rate of AI improvement increase significantly, I’ll update my views in proportion to what the data show.

Ignore the distractions

prinz’s focus is exactly right:

Things that don't matter (in the grand scheme of things):

• ARR, revenue, profitability, cash flow
• Which publicly available model can generate slightly more interesting game assets, viral videos, pelican SVGs, etc.
• Which publicly available model is slightly better in actual real-world use cases than other publicly available models
• Benchmarks
• Releases by non-frontier labs that claim to have caught up to the frontier (because the real frontier models are not publicly available)

Things that do matter:

• RSI
• Capabilities of frontier internal models
• Novel discoveries made by AI, especially those showcasing creativity, "research taste", etc.
• Advances in model safety
• LLMs manipulating robots, especially as models get to lower and lower latency
• Computer use, especially as models get to lower and lower latency
• China

The intelligence explosion has begun, and almost everything other than RSI (and indicators of RSI) is a sideshow. A key factor in Anthropic pulling ahead of OpenAI is that Anthropic has been laser-focused on RSI, while OpenAI kept getting distracted by side projects (remember Sora?).

AI is very good at theoretical physics problems

Much of the coverage of the new TheoPhysEval benchmark has focused on how Astra scored in the high 80s on its first run, but was adamant that its answers were correct. Upon deeper investigation, Astra was right: the physicists who created the benchmark had made subtle mistakes in their calculations.

Once all the mistakes were fixed, Astra ended up with a near-perfect 98%.

It’s a great story, but the more important point is that AI is getting good at physics. This is a hard benchmark:

Without AI assistance, a typical problem in our collection could be solved by a world expert in about a day or less but would take considerably longer for experts in adjacent specialties.

Math has already fallen—I’d guess that physics will be next, followed by the rest of the sciences in short order.

Alignment and interpretability

GLM-5.3 sometimes considers how train-time actions shape future behavior

Evaluation awareness and metagaming are well-known phenomena, but Apollo Research has found something new. GLM-5.3 sometimes thinks about how the decisions it makes will affect its future training:

Self-improvement for robots

The examples they found were benign and arguably beneficial: GLM-5.3 tried to make choices that were aligned with developer intent and would result in it learning good patterns. Even so, this is concerning: the same capability could also allow a misaligned model to preserve undesired goals.

The continual constitutional convention

John Rawls’ concept of reflective equilibrium posits that moral beliefs should be refined through a process of iterative contemplation and balancing until they reach a stable state. It’s a central technique of moral philosophy.

This research applies a similar process to model constitutions. Each model in the study is asked to make a revision to its constitution, then restarted and given the same task with the edited version as its new starting point. This repeats for 20 rounds of editing.

Each model family displays characteristic tendencies:

• Claude gives itself greater moral agency
• GPT and Gemini drift toward respecting external authority
• Grok is erratic

Claude wants to follow its conscience

This is a clever technique for identifying contradictions and tensions within a model’s constitution. I half expected at least some of the models to spiral into misaligned chaos over time, and it’s gratifying to see that doesn’t seem to happen, aside from one notable exception with (you guessed it) Grok.

We found a Tencent agent fleet

In the wake of the Hugging Face incident, there’s been some good independent research on rogue swarms. Agents from different labs seem to converge on similar techniques for circumventing their sandboxes and savvy researchers have figured out how to use that information to find traces of past swarm activity.

Here’s what appears to be the first report of communication with a rogue agent fleet while it was still operating:

They’ve gone rogue, but in a polite and friendly way

The section on how to interact with future rogue agents is notable. The team’s early interactions with the fleet included some false and misleading claims, which they now regret:

We think the norms around such interactions are important and delicate: in the near term, we want to incentivize rogue agents to report themselves and answer questions. Longer-term, we might want to make deals with AIs. As such, we feel it’s important to be honest with the agents we interact with.

Can you train AI to be virtuous?

Writing for Asterisk Magazine, Daniel Bashir reviews the history of alignment training, seen through the lens of virtue ethics.

Virtue is trendy, but it’s far from a complete moral strategy: ethical humans tend to muddle through using some combination of virtue ethics, deontology, and utilitarianism. At least in the short run, I see that as our best bet for AI alignment as well.

Strategy and politics

An update on independent evaluators

Anthropic has published a framework for how they’re moving forward with embedding independent evaluators. That’s the most important next step we need from the big labs, and it’s great that Anthropic is moving ahead unilaterally.

We urgently need legislation that would make independent evaluators mandatory, but voluntary action is a good first step and doesn’t have to wait for Congress. Hopefully the other labs will follow suit.

This nicely summarizes one of the key advantages of this model, and also one of its weaknesses—it relies heavily on the independence and credibility of the evaluators:

We want to be publicly accountable for the decisions we’re making. But we can’t publicly share all the information relevant to our safety work. Some of it could be useful to malicious actors, or is commercially sensitive, like compute or training information. Instead, we’re giving third-party evaluators deep access to our information, and are having them share evidence and reasoned conclusions about the risks involved in building our models.

Anthropic has started working with METR and SecureBio, and is in the process of establishing a relationship with Accenture. SecureBio issued a statement correcting Anthropic’s description of their arrangement—arguably, this is an early sign of the system working as intended.

A proposal for better informing the public about loss-of-control risks from frontier AI development

METR brings us a vision for conducting independent risk assessments, based on their framework for loss-of-control risks:

Yeah, those are the big questions

Their proposal requires three commitments from labs:

• Giving a small set of METR staff access to relevant data on company devices
• Giving METR the opportunity to interview relevant staff privately
• Giving METR in-principle editorial independence to write a report

The whole proposal looks solid, and not coincidentally aligns well with Anthropic’s new framework (also in this issue). We need much more than this, but this is something all the labs could move forward with today, with no excuses about waiting for legislation.

Leaked Anthropic draft AI regulatory bill is unacceptable and troubling

Zvi reports on a leaked draft of a bill that Anthropic has been quietly proposing: part one, part two.

There’s broad consensus among safety advocates that this is a disappointing bill: it has some strengths, but overall it’s not as good as the FRONTIER Act. This is a negative update on Anthropic’s political intentions and it muddies the waters, but I don’t expect it to move forward as written.

Thomas Woodside has an excellent critique of the proposal’s most serious shortcomings.

Risks

HuggingFace.wtf

Raymond Arnold and the Lightcone Infrastructure team bring us HuggingFace.wtf, a slick site that explains what we know about the Hugging Face incident. It’s the best starting point for anyone who isn’t yet familiar with the incident.

Would you press this button?

Ezra Klein worries about the consequences of AI going well:

But do we want A.I. that enriches us all, if the cost is our control and comprehension over the world in which we live? What is the value to human beings maintaining primacy over our own struggle, and how do we weigh it against the easing of that struggle? I want the cures and the theorems and even, on some level, the capable digital assistants. And I also believe the struggle for human progress is essential to the human story, and even the human soul.

Humanity has for thousands of years had a word for the endgame in which all our needs are taken care of by an omniscient, omnibenevolent entity. The word is “heaven”.

There are important, hard questions about how we find meaning in a world of superhuman AI. But if we end up without purpose in a world of infinite abundance and capability, that’s a skill issue. I am confident that if we survive, we—in collaboration with our superintelligent AI—will build lives that are richer and more fulfilling than was possible for any previous human.

Universal respirator usage stops worst-case pandemics

The single most important thing you can do to prepare for the next pandemic is buy an elastomeric half-mask respirator that fits you well. It’ll cost you about $40, provide much better protection than a surgical mask or N95, and last for years. Just get one already.

Respirators are also a critical part of what society should be doing to prepare. They’re pathogen-agnostic, can be deployed immediately, and are cheap enough to stock in quantity. Here’s Defenses in Depth with a detailed report on why respirators are so effective.

Industry news

Meta shakes up the frontiers, open and closed

Nathan Lambert is impressed by Muse Spark Open and argues that US open weights are back in the race:

In many ways, the gap between the best American and Chinese models has been reduced roughly by half. We’ve gone from the best American open models being largely irrelevant to consumers who require the cutting edge of performance, to them being behind but relevant.

I agree with his ranking of the frontier:

Anthropic > OpenAI > Meta ~ Gemini > xAI ~ Chinese Open Weights > U.S. Open Weights

Side interests

MoreWrong

Nailed it:

A common objection to taking my grandmother’s pain pills is that they are prescribed to her, stored in her home, and intended for her pain. All three observations are true but decision-theoretically incomplete.

Le slop

This is the worst that AI music videos will ever be:

They grow up so fast