ChinAI #375: Critiquing Anthropic’s Variant of Pacing the Frontier

Greetings from a world where…

“the highway up to Heaven got a crook on the toll”

…As always, the searchable archive of all past issues is here. Please please subscribe here to support ChinAI under a Guardian/Wikipedia-style tipping model (everyone gets the same content but those who can pay support access for all AND compensation for awesome ChinAI contributors).

Critiquing Anthropic’s Variant of Pacing the Frontier

A few weeks ago, my favorite blogger Dario Amodei, who moonlights as the CEO of Anthropic, published a new post titled “We Must Pace the Frontier.” This essay echoes the goal of “pacing the frontier”: a statement by over 1,3000 frontier AI company employees that calls for an international effort to “deliberately pace the frontier of automated AI development.” But, as always, Dario and Anthropic push a China-flavored variant of this goal. Specifically, Dario’s version argues that pacing the frontier also means keeping the U.S.’s AI lead over China “as large as possible” — a notable departure from the pacing the frontier statement signed by a lot of Anthropic employees. Let’s dissect three of the key assumptions in his latest post.

First, Dario assumes that if U.S. frontier AI labs slow down, then China will race ahead. This mirrors the “What about China!” argument that President Trump made when he downplayed the need for AI guardrails. Yet, as Matt Sheehan pointed out in his excellent conversation with Ezra Klein, “China has had the world’s strictest, most comprehensive, most burdensome A.I. regulations on its companies for three or four years at this point in time.” Most of these are not focused on existential risks, and I’d quibble with how burdensome some of these measures really are (e.g., last I checked, companies can submit self-assessments of safety/security for the algorithm registry application, as opposed to independent, third-party reviews), but Matt’s larger point holds. Indeed, one could argue that the Chinese government currently implements more guardrails for AI than the United States government, especially after the Trump administration’s revocation of the Biden administration’s executive order on AI safety.

Also, there’s some logical tension between this argument and Anthropic’s indignation over distillation. For the sake of argument, let’s assume Dario’s claims about distillation are true: it is a key driver behind how Chinese competitors catch up to U.S. frontier models. Well then, if U.S. companies slowed down their development of leading models, then this would necessarily also slow down the pace of Chinese frontier AI development, since they would be distilling lower-quality models at a slower rate. Another measure Dario advocates is to “strengthen security at the AI companies and prevent model weight theft.” Again, supposing that model weight theft is how Chinese companies can race ahead, if Anthropic slowed down, then that would necessarily slow down Chinese competitors.

The second assumption I want to unpack is Dario’s claim that if the U.S. can impede Chinese AI development by cracking down on distillation/model theft/AI chip exports, then these measures “would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important.” Throughout the piece, Dario keeps using the phrase “AI lead” or “Chinese lead in AI” without clearly defining what he conceptualizes as AI leadership. To him and many in this space, I think it’s a vision that one day Anthropicopens up a box and we get a superintelligent genie — they would probably name it Theos — that solves everything and immediately tips the global power balance. In line with this view, Dario asserts, “I believe that AI could cure most major diseases in the next 5–10 years.” Four quick rebuttals:

A. It’s pretty funny to see how Dario’s timeline for when AI becomes geopolitically most important has shifted. I traced this in my previous critique of “Anthropic’s Dogma on US-China AI Competition.” In January 2025, Dario told us that transformative AI would arrive in 2026. Then, in May 2026, Anthropic assured us that AI’s decisive impact on global leadership would occur in 2028. Now, Dario predicts that “the window when AI becomes geopolitically most important” is the 2029-2031 period.

Somehow, the goalposts keep moving further back even though the frontier lab leaders’ go-to argument against the slow diffusion camp is the possibility of recursive self-improvement. In fact, Dario notes that “since roughly this summer, AI has been advancing drastically faster” based on AI’s growing ability to build the next generation of AI. Again, notice the logical inconsistency. If you saw more clear evidence of recursive self-improvement just this summer, why would you further postpone the predicted date on when AI has a decisive impact on global leadership?

B. In my book, Technology and the Rise of Great Powers, I actually define what AI leadership means: a diffusion marathon over which country can more intensively and effectively adopt this general-purpose technology (GPT) across a wide range of economic sectors to achieve a significant productivity boost. Historically, this is how general-purpose technologies like AI have decided the rise and fall of great powers. I argue, “If AI, like previous GPTs, requires a prolonged period of gestation, substantial productivity payoffs should not materialize ­until the 2040s and 2050s” (p. 190). The path between initial discovery and diffusion into widespread use is a long road. Malaria is a curable disease, but 600,000 people still die from it each year.

Although, hey, maybe I just need to wait a few years before Dario and I will eventually agree. After all, extrapolating based on how the goalposts have shifted in just the past year and a half, his September 2029 blog post should confidently predict that AI’s greatest impact on geopolitics will occur in 2041!

C. Do not forget why these timelines and debates over the meaning of AI leadership matter. If you subscribe to my GPT diffusion view, then not selling powerful AI chips China will be counterproductive. By that time, Nvidia’s monopoly will fade, and all this policy will do is undercut U.S. companies. I wrote about this in detail in ChinAI #280.

D. I’m sorry, allow me one last point on Dario’s disease prediction. It reminded me of this line from my book (p. 187-188), where I reflect on the hype that surrounded nanotechnology:

Third, Dario’s emphasis on global formal agreements overlooks how meaningful U.S.-China cooperation on AI safety could occur through lower-level efforts such as technical exchanges and industry associations. Like Dario, OpenAI CEO Sam Altman also sees the Trump-Xi meeting this week as an opportunity to get a Nobel Peace Prize for a one-page AI agreement. Once again, history offers a useful corrective. Two of my articles support this point.

1. In my study of the history of U.S. nuclear safety and security assistance in the Cold War, an existing basis of technical cooperation between U.S. and Soviet scientists — including lab-to-lab exchanges and a Joint Verification Experiment project — laid the ground for one of the most meaningful formal frameworks (the Warhead Safety and Security Exchange), which could only be effectively implemented because of trusting relationships between both sides at the technical level (see screenshot below).

From my read, the “pacing the frontier” letter had zero signatories from Chinese labs. Organizations like Concordia and the Safe AI Forum are doing great work to highlight the many Chinese researchers who might also support that letter’s principles. I couldn’t care less about a grand bargain between Xi and Trump. It’s more important that we build up this basis for technical cooperation between people actually building and testing frontier AI models. It’s worth noting that this type of technical cooperation would be more obtainable if Dario and Anthropic hadn’t tanked their reputation with a large swathe of the Chinese AI community (see, for example, the controversy over Anthropic’s now-canceled tracker that secretly monitored Claude Code users in China).

1. The second transnational pathway to cooperation is through an international industry association dedicated to safety. My latest Review of International Political Economy traced how groups like the International Air Transport Association and the World Association of Nuclear Operators have helped China raise its safety standards in other high-risk technological domains. As a matter of fact, Dario’s post also recognizes this as an important case: “There is precedent for operating technologically complex, safety-critical systems millions of times without anything going wrong — for example, commercial airplanes — but it takes time to get it right.”

In sum, Dario and Sam are right that global governance of frontier AI will be a difficult challenge, but we can build on historical templates that are less dramatic but more meaningful than some Xi-Trump grand bargain.

As a coda of sorts, I want to reflect a little more on why I tend to single out Dario and Anthropic for criticism. To start, it’s always more worthwhile to punch up than punch down. In many ways, Dario’s statements frame the agenda for AI governance, and Anthropic is the world’s most valuable AI company. I also think it’s important to subject those views to scrutiny in a way that differs from lazier criticisms. I disagree with other Anthropic critics who dismiss these warnings of AI risks as mere hype or as a strategy to buy time as AI development stalls. I truly believe many of these researchers are genuinely concerned about the potentially catastrophic risks of rogue AI.

This belief is partly rooted in my engagement with the effective altruism (EA) community over the past decade, since joining the Centre for the Governance of AI as an intern in 2017. Back then, we shared an office space with the Centre for Effective Altruism, founded by William MacAskill and Toby Ord. Many of my colleagues and mentors from this community have moved to work in governance roles at frontier AI labs. Anthropic itself is a very EA-entangled organization. I admire the intellectual rigor and honesty of many people in this community, and I truly think that many of the Pacing the Frontier advocates are motivated to do the most good that they can. I also benefit from my EA-linked philanthropies, which have funded two of the grants that support my research at George Washington University.

At the same time, it feels wrong to see these AI company leaders claim moral authority when telling us how to govern AI. I’ve seen so many EAs try to justify having it all: you can work at a frontier AI lab, make tons more money, achieve an even higher impact on reducing AI risks, and wash away any guilt by donating a good chunk of your salary to good causes — just not enough that it would detract from a comfortable life (because if you one to worry about things like monthly bills, then that would take away from productivity and impact maxxing). Yet, power, wealth, and influence inevitably corrupt. See Sam Bankman-Fried.

It’s not just Big Tech’s corrupting influence. When one tries to both make as much money as possible and do the most good — hey, is it just me, or does that sound sorta similar to maintaining as large an AI lead as possible and also slowing the frontier? — there’s also a distancing effect. I’m reminded of one of my favorite passages in the Bible, from Luke 15:

Now the tax collectors and sinners were all gathering around to hear Jesus. But the Pharisees and the teachers of the law muttered, “This man welcomes sinners and eats with them.” (Note: tax collectors were socially ostracized for working for the Roman Empire; to the Pharisees, sinners were anyone who did not keep the strict religious purity laws)

Isn’t that something? Jesus did not just welcome the marginalized, the poor, and the sick but also ate with them. Contrast this with this high-stakes bet about AI’s impact on economic growth involving William MacAskill and at least two researchers who have worked or currently work at frontier AI labs. The five bettors put $240,000 into the pool. For context, the median annual household income in the United States is around $80,000. I don’t quite know how to put this in words, but something is lost when one tries to govern global AI risks without really being part of the world.

Dario and some AI company EAs position themselves as saviors of humanity, but I think they stopped eating with and welcoming sinners a long time ago.

Thank you for reading and engaging.

These are Jeff Ding’s (sometimes) weekly translations of Chinese-language musings on AI and related topics. Jeff is an Assistant Professor of Political Science at George Washington University.

Check out the archive of all past issues here & please subscribe here to support ChinAI under a Guardian/Wikipedia-style tipping model (everyone gets the same content but those who can pay for a subscription will support access for all).

Any suggestions or feedback? Let me know at chinainewsletter@gmail.com or on Twitter at @jjding99

For one, it gets easier to imagine away messier processes like the diffusion and implementation of technology and adopt the AGI-as-genie mentality.