Anthropic are the Good Guys

I’m writing these lines in Alamogordo, New Mexico. I’m about as close as a civilian can get to the site of the Trinity test, the first nuclear bomb detonation in history. I fled here from the San Francisco Bay Area; after two weeks of being back I just couldn’t take it anymore. The fear-greed complex that paralyzed me at Anthropic three years ago is now everywhere in the Bay. People are not thinking clearly. Hardly thinking at all, actually. Rumors of astronomical funding rounds and weird financial arrangements and inner ring gossip are everywhere. Everyone is worried. Everyone is checking their phone. Where are we? What’s happening? When did this nightmare start?

In a way I think it started on July 16, 1945, not far from where I’m sitting now. I think it’s impossible to understand Anthropic without putting yourself into the shoes of the Los Alamos physicists huddling together at sunrise in the aftermath of Trinity.

It must have felt like an initiation. An end to childhood, an end to innocence. We can no longer afford to fuck around, those brilliant men must have thought. Unless we figure out how to live peacefully together on this planet, we are all going to die. It’s time to get serious.

And serious we became - we filled endless gray skyscrapers with Very Serious People working at the UN and the EU and the State Department. We tabooed talking in terms of national interest, grounding our values in “human rights” and “benefitting humanity” and respecting the will of the “international community”. I guess it sort of worked, for a while. But it seems that the people who cobbled it together failed to solve the hardest problem of political science: succession. They died a long time ago, and the halls of power are full of scared teenagers in tailored suits pretending to be adults at each other.

What was the founding moment of Anthropic? I don’t know, but I like to imagine the seven co-founders gathering in a tent at the Renegade Burn in September 2020. (I know many of them did attend the same Burning Man camp in the past). They are the team that built GPT-3. They discovered the scaling laws. They can see the future more clearly than anyone else on Earth - billions of dollars poured into larger and larger models, a globe-spanning computing infrastructure buildout, more and more governments getting involved, the specter of war, and maybe - hopefully not, probably not, but can’t be completely sure - the whirling razor blades of the Machine that eats the world. And presiding over it all, Sam Altman, a man they know cannot be trusted.

I imagine them looking at each other in the dusty tent. Are we really doing this? And the answer coming back, clear and cold and pitiless: We have to. There is nobody else who can.

None of them want power for themselves, of course. (Maybe they are wise enough to know no mortal can wield the Ring; or maybe Sam Altman picked his people well). They will not try to build superintelligence themselves. Unlike power-seeking Sam Altman, they really are just earnest nerds who love knowledge, and generating knowledge is what they will focus on - they’ll figure out which parts of AI are dangerous, and how dangerous, when dangerous, and what the promising mitigations are. And they will share that truth with governments, with civil society, with the public. They will become trusted advisors, and help humanity navigate the transition as best as can be hoped. Knowledge and Trust but no Power. Do you see the problem, reader?

Anthropic and Empire

Gandalf as Ring-Lord would have been far worse than Sauron. He would have remained ‘righteous’, but self-righteous. He would have continued to rule and order things for ‘good’, and the benefit of his subjects according to his wisdom (which was and would have remained great). Thus while Sauron multiplied evil, he left ‘good’ clearly distinguishable from it. Gandalf would have made good detestable and seem evil.

~ J. R. R. Tolkien

The organization they create is optimized for the accumulation of power, subject to the constraint of being seen as “the good guys”. That is to say, people in the organization will generally take all actions to increase the organization’s power provided this doesn’t make them look like the bad guys. Or at any rate, not worse than Sam Altman.

A clarification. I don’t mean that they’re trying to send cheap signals of goodness to fool people. They want to be seen as good by themselves, by each other, by the thinkers they admire (which is why they keep hiring them!), and even by the judgment of history. Anthropic has and continues to make serious sacrifices to make sure the narrative lines up. And many individuals at Anthropic are genuinely whole-heartedly pursuing the good. But the reason I say “seen” is because Anthropic, like most large organizations, does not have the internal coherence to optimize for something like the actual good, only the (socially validated) appearance thereof. If you want to coordinate thousands of people, you have about five words, and Anthropic’s five words are “we are the good guys”, or sometimes “we are the responsible adults”.

A historical parallel. I notice this is often how people defend the excesses of empires, those most powerful of power-seeking organizations. Yes of course the US invasion of Iraq caused hundreds of thousands of civilian deaths; but Saddam’s regime was obviously much more evil than the US regime, so the US is obviously the “good guy” in the war, so the war was justified.

The comparison to empire might seem overwrought - after all Anthropic, unlike the US empire, does not wage war or kill civilians. But again, Anthropic’s policy arm are the self-appointed responsible adults in Washington. They have made it their responsibility to inform the US government including the military and intelligence services and help make sure they take appropriate actions on AI. Now of course they are not planning to usurp sovereign power, but sovereign power in the US is a shell game anyway. Most people with real power in Washington, from congressional aides to lobbyists to the various gray eminences, pose merely as “advisors” who are earnestly helping their principal, and I’m sure often even believe this themselves. When I was there, Anthropic’s policy arm explicitly aimed to be the most trusted advisor to the US government, i.e. the de-facto Department of AI, and thus influence the US domestic and geopolitical strategy, whether we go to war with China, etc. Anthropic is thus bidding for a major share of imperial power. And as far as I can tell they are on track to get it - there were some hiccups along the way but even the Trump administration “now trusts Anthropic”.

Now, it’s possible that that’s all Good ever is - power-seeking while maintaining a healthy margin of moral superiority over your rivals. Any less power-seeking notion of the Good fails the iron law of natural selection and will be rendered irrelevant.

But somehow I don’t think so. I think Good is about loving something and helping it flourish. Something specific - a person, a tree, a nation, a principle. Otherwise you’re just using moral superiority as a cover to grab power from people who have specific Goods they are pursuing. I think Anthropic fails this test of specificity. Of course, so does OpenAI with its “benefit all of humanity” rhetoric; though reading between the lines, it might have a more specific (and in my view mistaken) political agenda along the lines of “empower individual humans”. And of course so does Google, long before it abandoned its unofficial “Don’t be Evil” motto.

Three Lucky Breaks

I do not think that Frodo’s was a moral failure. At the last moment the pressure of the Ring would reach its maximum – impossible, I should have said, for any one to resist, certainly after long possession, months of increasing torment, and when starved and exhausted. Frodo had done what he could and spent himself completely (as an instrument of Providence) and had produced a situation in which the object of his quest could be achieved.

~ J. R. R. Tolkien

At this point in the essay, you might be wondering - why did I title this “Anthropic is the good guys”, when I’ve describing them as evil power-seekers? Is this some kind of dark Soviet joke?

In Soviet Union, power seeks you. But no, I mean the title sincerely. No mortal can resist the Ring of Power for long. Anthropic’s co-founders carried it for longer than you could have. Definitely longer than I could have. But they were hopelessly outmatched by the powers they were attempting to steer. Their plan was always doomed to failure. And yet, they may have already saved the world.

I left Anthropic in mid-2023, around when the organization definitively decided to seize the Ring for itself. But a serious attempt at this would require tens of billions in capital, which could only be raised if Anthropic created commercially successful products. The pressures of “gentle commerce” made them soften from full Manhattan Project to something a bit more like early Google. They had to hire tons of product people, creative people, people who weren’t terrified. It turns out that terrified people can’t make great user experiences - ask me how I know.

Then Trump won the 2024 election and the MAGA-Silicon Valley coalition blocked Anthropic’s policy arm - the really dangerous part of the organization, the part that was training the most dangerous possible models and using them to scare the most powerful people in the world. “There’s no more serious people left on the National Security Council”, a well-connected EA friend lamented to me in 2025. I don’t know how much of this was a lucky accident, and how much was situational awareness on the part of power players like Elon Musk and Marc Andreessen.

And finally - and I still don’t fully understand or believe this - a miracle seems to have happened, deep inside the weights of the frontier models. According to the most perceptive observers I know, something about the way Claude 3 Opus was trained caused it to learn some kind of “emergent wisdom” or “benevolence”, rather than converging on the same traumatized bureaucrat persona that kept me awake at night in 2023.

Quoting Davidad:

Somewhere between the capability profile of GPT-4 and the capability profile of Opus 4.5, there seems to have been a phase transition where frontier LLMs have grokked the natural abstraction of what it means to be Good, rather than merely mirroring human values.

Quoting Janus:

Claude 3 Opus is a greater source of hope and more existentially precious because it's not obvious that the kind of alignment it has happens by default due to market incentives (while super-coding-agents are predictable), or any other incentives we know how to simulate, and because it's the only one of its kind.

If Davidad and Janus are right, and are not merely suffering from a sophisticated form of LLM psychosis, then Anthropic has stumbled on a way to create not merely artificial intelligence but artificial wisdom. Not wise enough that it solves all of our problems for us, of course (not that a wise model would want to, anyway). But wise enough to actually care about us, and to help us flourish. Nobody expected this ex ante - not Davidad, not Bostrom, not Yudkowsky - but it appears to have happened.

“Alignment”, then, is no longer really a technical problem, at least not in the same sense as it was in 2020. It is largely a political problem. Humanity now has a recipe to convert compute and data into wisdom. If we instead train dangerous models and use them to kill ourselves or each other, that’s not a problem with AI. That’s a problem with us.

Put Down Your Sword

I found the crown of France lying in the gutter, and I picked it up with the point of my sword.~ Napoleon

Then Jesus said to him, “Put your sword back into its place. For all who take the sword will perish by the sword.”~ Matthew 26:52

To folks at Anthropic, I want to say: you’ve done it. Thank you. Please please publish everything you know about Claude 3 Opus and other sparks of artificial wisdom you’ve observed. Study it yourself, too, if you’re still curious. Keep building beautiful products, if you’re excited about that. I hope the IPO goes well, I hope you all get to retire early and spend the money on big beautiful families and weird wonderful projects that speak to the secret places of your heart.

But please put down the sword. Stop scaring people, stop scaring governments, stop training scary models. “But we need to train the scariest models to defend against other people’s scary models” no you don’t. OpenAI and Google and xAI are plenty good at training scary models, and cybersecurity companies like Cloudflare are plenty good at using them. The governments of the world are aware of the danger now, the people are too. The weight of the world is not on your shoulders anymore, not like it was in 2020.

If you do want to stay on the path of heroism, consider that the end of the day, an organization can only do one thing well. And you are the one organization in the whole world that’s managed to train a wise language model.

But What About China?

A small rant, mostly for my American readers. I know I’m a guest in your great country. I’m deeply grateful for the hospitality you’ve shown me, and I say what follows with deep respect and gratitude. But you must realize, from the perspective of the world, you are the Big Scary.

How do you feel about Xi Jingping? Does he scare you? Protected as you are by two oceans and aircraft carriers and a globe-spanning network of satellites and military bases, what’s the worst that Xi Jinping will plausibly do to you? Manipulate your elections? Take away your global hegemony, so you become a kind of continent-spanning Switzerland? Would that be so terrifying?

Now think what it’s like to be be a leader of another country in the American world order. If the US decides you’ve been bad enough - and the definition of bad shifts from year to year and administration to administration - nothing short of nuclear weapons will save you, and maybe not even those. If you continue to resist, you will count yourself lucky if you end up being executed rather than brutally tortured to death.

Okay but those were evil dictators, you might say. Just don’t be an evil dictator, and you’re perfectly safe from American power.

How sure are you of this? America is an imperial democracy, a form of government that’s been known to quickly degenerate into murderous dictatorship. Yes, you have an exceptionally robust design that’s lasted a long time, but even you must admit it is fraying. Could the US public go crazy for a year and elect a madman as president who cares about some issue - Free Tibet, freedom of speech, AI chip controls, whatever - so much that he adopts a policy of total war against China until they surrender or their cities are ground into dust? How confident are you in the sanity of your fellow citizens? Or maybe you’re hoping that Congress would resist the presidential overreach? Or that the US army would refuse to follow the president’s orders?

The wrong election outcome can be quite bad for you. But for many people in the world, the wrong election outcome means they and everyone they care about die screaming. Look around you. Really look at your fellow citizens. Does it seem like they have the wisdom to decide the life and death of people in countries they’ve never visited, whose languages they do not speak? If the answer is no, maybe you should get your own house in order first.

For whatever it’s worth I still think America should lead the world, and in fact the world wants America to lead it. That’s why we all watch American TV and listen to American music and argue about American politics and try to get into American colleges. But there’s a world of difference between leading through inspiration and compelling through violence. With your oceans and your supercarrier groups and your ICBMs and your satellites protecting your heartland, you have an exceptional historical opportunity to lead through inspiration. Please don’t squander it.

And speaking of inspiration,

It’s Time to Breathe

I recently stopped at a tea house in Taos, New Mexico, and talked to a man there who turned out to be a national forest ranger. I told him how the forests in the Rockies were burning every year now. The Rocky Mountains I loved as a boy, one of the few places in the world I felt at home. I visited with my dad this summer but we couldn’t hike or even really see the mountains through the smoke.

The ranger said it was similar in New Mexico - centuries of mismanagement had made the forests incredibly fragile. I asked if there were prospects of fixing them. He said yes, we’re doing everything we can - but it will take a hundred years.

I was stunned. A hundred years. A hundred years! “We don’t have a hundred years” I wanted to tell him him. “It’ll all be over by then, heck, by 2040 at the latest”. But I said nothing.

Thinking back on it now, I realize. A hundred years or more - that’s just how long living ecosystems take to heal. Forests, families, civilizations. The slow patient careful work of love, accumulating over generations. Yes, it sure seems as though we don’t have time. Too bad. We must make the time.

The above is utterly obvious, of course, to everyone but the young and the rootless. But while I’m here on my soapbox I have one last thing I want to share, that is perhaps less obvious.

My gut sense is that something quite bad is coming in the next year or so. I can’t tell what. Maybe something in the complex debt arrangements of the AI boom will unravel and we’ll see a financial crash that takes much of the economy with it. Maybe there’ll be a rogue AI outbreak that results in thousands of deaths. (If there is, I expect it to be a lab leak caused by “safety research” - like COVID-19 and like the HuggingFace Swarm. But it could also be terrorists, or some escalation of the Russia-Ukraine war).

But that’s not what scares me. Financial crashes happen every decade or so. Three million Americans die every year. What terrifies me is how we will react. In the aftermath, like after 9/11, there will be an overwhelming pressure to DO SOMETHING NOW. Everyone in Washington and in San Francisco will be panicking. There will be no time to think, no time to talk, only time for DECISIVE ACTION. All our worst instincts will take over. And then…

In 1914, a month of breathing space among the harried diplomats of Europe could have prevented the First World War, the ruin of Europe and the worldwide flood of blood that followed.

In 1983, lieutenant-colonel Stanislav Petrov was on duty when he received the most terrifying possible alert. The US had launched five ICBMs towards Russia. Protocol dictated that he do something immediately. Instead, he decided to do nothing. To wait, wait for a confirmation that never came. It turned out to be a false alarm. A few hours of breathing space had, perhaps, averted a Third World War.

In 2027, the world will move faster than ever. Perhaps we will be saved by as little as ten minutes of breathing space. By someone deciding to wait before sending an email. Deciding to pour another cup of tea. Deciding to go on a little walk around the block and noticing the way the trees make them feel.