ModelTalk: Pacing the Frontier

ModelTalk regulars , who writes Interconnects, and writer join to work through a week in which Dario asked the industry to slow down and Trump called the whole thing a conspiracy.

• Why the Hugging Face hack broke through where years of extinction talk didn’t, and how Bernie, Tucker, and Rogan ended up on the same side
• What “pacing” even means: Nathan’s case that nobody can measure the frontier, and why it still beats “pause”
• Who gets Slack access: METR’s monopoly on trust, CAISI’s $10 million, jury duty for AI researchers, and why nothing survives without legislation
• Will China slow down too? Zhipu and Moonshot turning the crank, military secrets leaking into Claude through proxy routers, and the MSS waking up
• The Sacks clap-back: whether product liability does the job, and what an eighteen-month pause costs in cancer cures and vibe-coding projects
• Send in Demis! plus Lu Xun as the antidote to Chongqing TikToks, and what DeepSeek’s Chinese name owes to Qu Yuan

Listen now on your favorite podcast app.

The Safetyists Break Containment

Jordan Schneider: Ladies and gentlemen, ModelTalk is back. Where do we start this timeline?

Nathan Lambert: Hugging Face, OpenAI. July.

Jasmine Sun: There’s the pacing letter at the end of July, then Hugging Face shortly after that, and everything has been escalating since. Then the new reports that Hugging Face was so much worse than we thought. And the Jacob Coxon post is interesting because it broke containment: Sheryl Crow and Maggie Rogers are posting about it, and he’s hanging out with Bernie or whatever.

What’s most interesting to me is that I had done this data centers piece, and in mid-August the Hugging Face incident had shown up on people’s radar, but it wasn’t huge yet. At the time people were freaked out about data centers in a really big way, and all the politicians were talking about that. Some of that general, diffuse, populist anti-AI sentiment has now collided with, “Yeah, also the safety stuff is super real now. We are really worried about these misaligned agent swarms.” The AI safety folks and a broader public that was already souring on AI — souring on data centers, souring on the companies — have collided into a mega moment for AI safety and regulation. That’s how I’ve been tracking it, since early August probably. What do you guys think?

Jordan Schneider: Do the sine waves all hit now — Daniel Kokotajlo on Joe Rogan, Nate Soares on Tucker Carlson?

Nathan Lambert: Soares on Tucker is way more crazy.

Jordan Schneider: Was this inevitable? Was it happenstance? Is there a grand EA media comms conspiracy? How contingent is the past week we’ve just seen, Jasmine?

Jasmine Sun: I was wondering for a long time whether these two sides would collide. There are internal debates within the EA community about whether the frontier safety people should engage with people like Tucker Carlson, who is seen as not aligned with them on a bunch of other issues, and maybe not an epistemically rigorous, truth-seeking guy. In general there was a lot of hesitation from the more technocratic wings of AI safety about whether to talk to Bernie, whether to talk to Tucker, whether to try to make this a gigantic mass movement or whether that would be destructive.

I wasn’t sure it was going to happen, honestly, because for a long time the left and the people who didn’t trust AI people also didn’t trust the safety people, who were seen as part of the same Silicon Valley tribe of SF and Berkeley weirdos. So it’s surprising, but not beyond imagination, that the coalition did emerge. The left got over its distrust of AI safety and its belief that maybe AI was fake. Bernie probably did more than anybody to wake up the broad populace and the left to think, “Shoot, AI is real, AI risk is real. This isn’t a fake marketing tactic made up by the companies.” We’ve seen a phase shift there.

And then the public got so riled up about data centers, plus a bunch of other simmering anti-AI stuff, that when Hugging Face hit, it broke through just enough that they decided to care about actual frontier safety stuff. At that point, if you’re working on frontier safety, why not build that coalition?

Nathan Lambert: Do you think the public actually cares about it? I feel like the public was like, “Huh, that’s weird.”

Jasmine Sun: People care. It’s not people’s top voting issue right now. I don’t think anyone’s going to vote on AI as a top-three issue, or even top-five. But there is a sense that these companies are being reckless and need to be regulated, and I think that now extends to frontier safety in a way that it didn’t before. If you ask the public whether we should have embedded third-party evaluators, or whether we should fund CAISI, no one’s going to have opinions on that. But the general idea — these companies are out of control, there’s way too much power concentrated in them, we need to regulate the data centers, we need transparency, we need government auditing, we need to investigate this AI crime — I think that’s a pretty common intuition. We heard AIs are going rogue. Is that okay? Send in the FBI.

Jordan Schneider: There’s also something about the fact that the OpenAI–Hugging Face hack is so narratively compelling. It fits very cleanly onto all the other media that has been created over the past forty years about rogue AI. So when the New York Post cover is the Terminator, it’s not that far off from what actually happened. The facts of the incident rhyme enough that a median voter looking at it from a headline standpoint can pattern-match it to things that are legibly scary.

Look, it’s Rosh Hashanah on Saturday. You’re not supposed to be reading the news or on your phone; it’s a no-electricity day. And the number of questions I got at synagogue — “We’re going to pace the frontier, Jordan? What does this all mean?” My mother was asking me if she should withdraw some money and hide it under a mattress.

This is a multiple-orders-of-magnitude breakthrough that some boring industrial safety incident — a chemical leak from a plant into a river — would not produce, because you can’t tie that to something that sounds like Skynet. This, you can.

Sci-Fi, Non-Dismissively

Jasmine Sun: On Hard Fork’s Hugging Face episode, they were talking about how some people critique calling the Hugging Face hack “sci-fi” because they think it’s dismissive. But actually it is sci-fi, in a non-dismissive way. It fits cleanly into all the sci-fi narratives about rogue AIs, and so it’s, “Shit, the sci-fi is real. All the things from the Berkeley people that sounded like science fiction really are that bad, maybe.” And the message board thing.

Nathan Lambert: I feel like that’s a big step from what the AIs did to all the other fictitious scenarios from the — what’s a better word? I don’t like using the doomer word. I’m not absolutist about it, but the safetyists —

Jasmine Sun: You can say safetyists. Safety people. That seems fine.

Nathan Lambert: Those people have a lot more extreme scenarios. Yes, the Hugging Face incident is surprising, but it’s also not that surprising if you think about how persistent one OpenAI model is. If you strip off the guard layers, have a thousand copies, and tell it to do a really hard task, I’m not that surprised it does some wacky stuff. It’s not surprising to me that the model told to hack then hacks. That’s why it’s a problem if you tell more extreme stories to people. But people like scary stories, and that’s why it traveled so far.

Jordan Schneider: Sure — you, Nathan Lambert, author of Interconnects, aren’t surprised. But people like scary stories. If you’re going from asking ChatGPT to convert a recipe from milliliters to pints to “by the way, this thing you thought was merely annoying because it wrote your kids’ homework for them can also hack into your bank” — I don’t begrudge people for being perplexed and upset about data centers, and I don’t begrudge people for being freaked out.

Nathan Lambert: I don’t think hacking into your bank is a big jump. I think the extinction of humanity is a very, very big jump. Even hacking into your bank would often get corrected so long as somebody notices; banks are very good at correcting for fraud and putting things back in the right place. It’s not great that the status quo for a few years is going to be more undocumented cyber incidents happening all the time, and hopefully they don’t cause a lot of damage. But I just think we should not get too escalatory.

Jasmine Sun: I don’t know whether the public is worried about extinction. Some folks like Bernie are worried about extinction, I think, in a real way. I’m not sure the public is worried about extinction specifically —

Nathan Lambert: Extinction wakes the public up. If you tell them repeatedly that everyone’s going to die —

Jasmine Sun: Yes. But people have been saying extinction for a while, and it was a real-life incident that woke people up. Obviously there’s nothing about newspaper editors that is representative of the American public. But one big thing from my experience talking to media people is that for a long time, mainstream media people would say all the safetyist talk is speculative. It’s about stuff that could happen: maybe these AIs will collaborate or deceive or engage in human-like motivations. It all seemed too speculative, or it was only happening in made-up sandbox environments. What changed for a lot of these people was that it was a warning shot. This thing actually happened in real life, rather than being a forecast. I don’t think it was the extinction talk that changed things. It was seeing another serious company like Hugging Face actually get hacked — not because of misuse by a terrorist or some bad actor, but because OpenAI could not control and monitor its own agents — and that happening in reality, not in a made-up red-teaming scenario.

Nathan Lambert: I agree it’s going bigger. I’m griping about the people who are pushing it. I hate the word “conspiracy,” because I don’t think the agents had a conspiracy. I’m fine with some of the anthropomorphizing, because it’s a useful lens for describing how the models evolve and act in an environment: the agent will be like, “I need to figure this out,” and then you see a bunch of actions trying to figure it out. So words like “goal” are pretty fair. But the whole language spiraled into civilizations and conspiracies, and it goes so broad that I’m just like, nah. I got snippets of this on my honeymoon. I’d get a text and be like, “This is dumb, man.”

Jordan Schneider: Three paragraphs in a red-team write-up is way less compelling than a company that thought it was being hacked by the Chinese and it turned out to be a customer — a company that’s supposed to be one of the most technologically advanced on the planet, which has no idea what it’s doing.

Jasmine Sun: And to other people, it’s two AI companies. It’s not some small business with bad cybersecurity that got hacked. One AI company hacked the other, and neither of them knew about it because —

Nathan Lambert: Don’t ask me about the cybersecurity of Hugging Face. No comment.

Jasmine Sun: What do you guys make of the discourse in cyber-land about how much this is an alignment issue versus a cyber issue?

Nathan Lambert: When I talk to cyber people, it seems pretty clear that we need mass preparation for the tools that are going to proliferate. There’s a clear cyber issue, which is that our infrastructure is not ready for the type of tool that’s going to be ubiquitous. And there is an alignment issue, especially at the frontier: the more differentiated a capability, the more responsibility there is to align the models. But we have seen that the frontier labs poke a hole in what is possible, and then in three to six months there are three models that can do it, and in a year there are twenty-plus organizations. So it’s an alignment issue for a transient period of time, but there’s a constant cyber issue that is not going away until we solve it.

Pacing, Not Pausing

Jordan Schneider: Let’s talk about frontier pacing. We’re not saying “pause” anymore like we did last season; I’m glad we got the rebrand. What were your impressions of Dario’s essay and the pickup we’ve seen so far?

Jasmine Sun: How do you define pacing, Jordan?

Jordan Schneider: That’s the question, right?

Nathan Lambert: One point of MMLU per year.

Jasmine Sun: It’s clearly not a pause, and it’s not slowing down. They chose a word that still implies one is moving ahead.

Jordan Schneider: Nathan, as the runner on this podcast, I think we’ve got to defer to you here.

Nathan Lambert: We don’t know how to measure frontier models. Any pacing would imply you’re trying not to go past certain measurements. It’s pretty self-serving, and OpenAI and Anthropic could do what they want with it. But I like it. The sentiment is very valid. These companies need to share more for us to have the ability to discuss this. I’m sick of hearing people at labs say, “You don’t see this. Internally we’re also scared by the pace of progress.” What does that actually mean? I don’t feel scared by these people, who have hyped stuff up a lot and some of whom are a bit out of touch. It goes over my head a lot of the time. I don’t know what to make of it without more transparency.

Jordan Schneider: If we’re moving marginally toward spending more money on safety and alignment and less on pushing the frontier at xAI, OpenAI, and Anthropic — and again, this is a really hard coordination problem — would it be horrible for the planet? Would it mean we lose to China? Probably not. There are lots of other levers that go alongside this if you’re stressed about the China angle, like all the compute control stuff we’ve been screaming about for years and years.

Nathan Lambert: I am interested in the China stuff, because there have been some interesting comments. But the pacing framing implies that it’s more metronomic and measurable than it is. It should be more of an industry standard that independent people are in all of the labs — clusters of people who can see into one or multiple labs — and agree that the labs have done sufficient safety work for the current capabilities of their model. Every model has different strengths and weaknesses. OpenAI’s models are so persistent that there’s a very different safety problem there. I don’t know if you’ve used /goal in Codex, but if you phrase your goal really specifically, it just goes and works for two days on it. And the Claude models in production that I use seem almost borderline lazy sometimes, where you need to put in more input.

Jasmine Sun: I find them lazy.

Nathan Lambert: That’s a very different alignment problem. I’d see pacing as: these companies have some of the same people in them who can see what’s happening and give approval — “Yes, we are satisfied. We know there’s uncertainty, but appropriate risk has been accounted for.”

Jordan Schneider: I don’t understand how this happens without the government being the one who says yes and no. Say METR is woke, we’re not going to trust them, they’re basically Anthropic — so it’s going to be METR Nathan Lambert. METR Nathan Lambert shows up and says, “This isn’t cool yet. You gotta bake some more.” And OpenAI says, “No, it’s fine, this is safe, what are you talking about?” Then what? Your contract times out —

Nathan Lambert: My contract times out.

Jordan Schneider: — you write a blog post saying this is fucked. And then what? Either the government trusts Nathan more than it trusts the company upon which the entire US economy depends, or…

Nathan Lambert: It’s just so hard to implement. Although there are all these academics saying we have untapped students and professors who would be great for studying all of this. And I’m like, how are you going to set this up? How is a university going to sign these NDAs with these companies and ship their grad students to San Francisco? In spirit, it’s nice.

Jasmine Sun: It’s a pretty crazy level of access. Dario is saying — and I think he plans on it — give the METR people desks and Slack and codebase access. On one hand, yes, I support having more than just METR, because METR is a very specific approach to AI safety, and it would be more credible if there were a few other orgs not from Berkeley. At the same time, I can’t imagine Anthropic being willing to let in some nonprofit from, I don’t know, Boston, and give them full Slack access, because we know how paranoid they are about security and how little they trust most people. The reason they trust the METR folks is that they’ve personally known them for five-plus years, and they’ve decided, yeah, Ajeya Cotra should get Slack access. We’d be okay with Ryan Greenblatt getting Slack access. I’m not sure that transfers to other people at METR, or to other orgs.

Then the second trust question is, as Jordan said: what does the government do if the third-party evaluator disagrees with the gigantic company the economy rests on? Does the admin trust METR, or whoever that other organization is? Right now METR seems like clearly the best we have, so I’m pro-METR. At the same time, when I saw Dario’s embedded-evaluators thing, third-party evaluators are almost the compromise position compared to having the Trump admin just send their guys in and start knocking on Anthropic’s doors to shut the models down. Bringing in nonprofits is actually the more pro-AI-progress option, because the alternative is the Trump guys showing up and saying, “You can’t release this.”

Nathan Lambert: Who’s going to do this? There are so few people.

Jordan Schneider: And we lived in that timeline, right? It didn’t stick. Maybe it didn’t stick because it happened a little too early, because the models weren’t quite as scary, or whatever.

Who Gets Slack Access?

Jasmine Sun: Next time around, let’s look at the incident again. Amazon says Fable is dangerous. Andy Jassy or whoever calls Trump, or Susie Wiles, and says, “We found a jailbreak. This thing is super unsafe.” The admin calls Anthropic: “Hey, Tom Brown, apparently your model is super unsafe.” And Tom Brown says, “No, no, no, METR says we’re cool.” What happens then?

Nathan Lambert: Andy Jassy is getting listened to a lot more than my friend Chris at METR.

Jasmine Sun: It’s Ryan Greenblatt. Ryan Greenblatt specifically.

Jordan Schneider: Or CAISI, which has ten million dollars and hasn’t been able to find a head four years into this whole thing.

Nathan Lambert: They only have ten million dollars? God. As somebody fundraising in the nonprofit scene, I feel bad that that’s a pretty small amount of money.

Jordan Schneider: It might go to thirty.

Jasmine Sun: And people keep trying to shut them down every six months, right? They barely get saved, and then someone tries to shut them down again.

Jordan Schneider: Nathan, who is going to work here? The most qualified people are people in the labs, because you really need model training experience to know what the heck is going on, presumably. Where would the money come from such that everyone would think it’s clean and trustworthy? And who would do this job?

Nathan Lambert: You just have to pay them what they’d get paid in salary and not have stock. You have to be paying people 400K to a million in cash a year. That’s what most of the nonprofits are converging to: the same or slightly more cash, no stock.

Jasmine Sun: I think that’s what METR pays. Cash comp. Which is fair, because there are people willing to take pay cuts to work at nonprofits; you just can’t take so much of a pay cut that you can’t get your house in San Francisco.

Jordan Schneider: But then where does the money come from such that people are comfortable with it? It should be an industry consortium, right? But if you’re worried about it not coming from AI safety donors, and instead it’s an industry consortium, is that not a captured thing that no one’s going to trust? That Bernie Sanders won’t trust?

Nathan Lambert: Taxpayers. I think the only way to trust is diversity in backgrounds and funding. Which is why — I don’t really know if I’d be good at this or accepted, but I’ll talk to people about doing this job.

Jasmine Sun: It’s government funding, right? Bank supervisors are government funded. They don’t take money from the banks; they take money from the government, and then the government licenses other people with more expertise to go sit in the banks and monitor them. And every six months or so you have to cycle people out so they don’t get ideologically captured by Goldman Sachs.

Nathan Lambert: We tax the data centers to fund CAISI.

Jasmine Sun: And then CAISI either manages to hire people or pays METR plus two other orgs, and mandates that Anthropic and OpenAI accept those people. So Anthropic can’t say, “We’ll take Ryan Greenblatt but not Nathan Lambert.”

Nathan Lambert: I would love that email. That would be freaking hilarious.

Jordan Schneider: There should be some jury-pool thing: fifteen people, you get five no’s.

Jasmine Sun: Oh my god, AI researchers just got mail. “You’ve been summoned.” That would be so good.

Nathan Lambert: I would do my best. I don’t feel like I’ve been so critical of Anthropic that they can’t let me in the office. I’ve been in the Anthropic office; they didn’t turn me away. It’s a very interesting social dynamics problem. It’s an industry concentration problem.

Jasmine Sun: I do think we just need legislation. Insofar as there’s no legislation, it’s voluntary, which means that even if the orgs are trying their best to be independent and aggressive, there is an extent to which METR or Redwood or whoever will not antagonize the org that is giving them voluntary access. You need some legislation, whether it’s the government licensing METR or the government sending its own people in. There’s just no world without federal involvement that I think is sustainable.

Too Woke to Exist

Nathan Lambert: You can also imagine a future where in six to nine months we’ve been pacing the frontier at xAI — SpaceX AI — Anthropic, and OpenAI, and Meta keeps crushing it but isn’t pacing. The independent evaluators are at those three orgs and have all found the same absolutely critical failure mode, but it’s all private information at the separate orgs, NDA’d at the evaluators, and Meta doesn’t have them and just YOLOs it and unleashes the bad thing. How do you deal with this?

Jasmine Sun: That’s why they’re working on the antitrust exemptions, right? Isn’t it because of that? Maybe you have thoughts on the antitrust exemption, Jordan, because it’s also framed around distillation and China stuff. But my understanding is that it’s really a frontier safety thing. There are a bunch of things where the labs can’t coordinate on safety because of antitrust law, so now they want to carve out an AI safety antitrust exemption. And the current bill is framed around beating China, as things usually are.

Jordan Schneider: Let me take half a step back. First there’s the question of what the legislation is. I had Astra pull me all the frontier AI safety bills for this Congress, and none of them do this in a compelling way.

Jasmine Sun: I think the Frontier Act is supposed to be the one licensing the orgs.

Jordan Schneider: Tiered requirements for model disclosures. But it’s basically “Commerce, you have the power to do this.” Once you kick it over to the executive branch, that’s still the dynamic we had when Fable got shut down. If the White House calls you and says, “Don’t fucking do this,” you’re probably still not going to do it, unless you’re so confident that you’re willing to go to court to fight it. That’s what Anthropic did in the Pentagon case, because they were going to try to shut down the company. But not in the Fable case, because there it was, “Okay, you’re worried about safety, we’re worried about safety, let’s have a conversation.” I’m glad we’ve at least memetically come to a good place. We can work with this as a starting point for a discussion, as opposed to us being too woke to exist as a firm.

Jasmine Sun: “Too woke to exist” is such a good line.

Nathan Lambert: Do we think OpenAI and Anthropic have these independent evaluators set up within three weeks?

Jordan Schneider: Who is it going to be?

Nathan Lambert: METR.

Jasmine Sun: It’s going to be Ajeya and Ryan, I think. Eventually they might do something else, but to start, it’s just METR.

Nathan Lambert: I think it should be institutional and not individual. It should not mention the people. It would be good to know how the Hugging Face–OpenAI investigation went down. If it was “METR, send your best four people, they’ll sign the NDAs and sit in the office,” that’s good. METR should decide that. But if OpenAI and Anthropic say, “We just want these three people,” that’s not good. That’s not a good way to do it. If METR has forty people, implicitly the labs are going to know who METR’s most likely to send, but there needs to be enough trust that the independent organizations can have a say.

Jasmine Sun: The thing with the ecosystem right now is that everything is so messy and flying by the seat of everyone’s pants that none of the things happening now are being done the correct and proper way. And everyone knows it. It’s not like anyone thinks this is better. Even the initial METR–Redwood investigation of the Hugging Face incident was completely voluntary on OpenAI’s part, and we can all criticize them for only giving access to seven days of Slack logs or whatever. We would all want more. At the same time, there was no legislation. No one was required to do shit. We were purely relying on voluntary goodwill, and the fact that they happened to trust these three random people enough to bring them in. So there’s a balance between what we’re trying to move toward in an ideal world and what we want to enshrine in legislation — and then, personally, also: this is what we’ve got right now. Okay.

Nathan Lambert: And this is in the face of rumors that the labs have such better models. Anthropic hasn’t released something in months — they’ve released 5.1, but they have other stuff cooking that is well stronger than this. So the US–China angle is that the US labs are taking longer and longer to release, at least at the frontier, and the Chinese labs don’t appear to be slowing down. But we don’t get good information out of that ecosystem.

Will China Slow Down?

Jasmine Sun: I’m curious for your take on the whole question of, if the US slows down, does Chinese progress slow down, and by how much?

Nathan Lambert: Within the current paradigm, I think the Chinese labs are set up to hill-climb very well. There was this Dwarkesh podcast with John Schulman, Charlie O’Neill, and Beren Millidge, and even they were talking about how GLM-5.3 is potentially better than Claude Sonnet 5 and Opus 5, just because it’s somewhat easier to use. Zhipu — Z.ai — has shown that their post-training stack is very mature and can hill-climb and make very good models. GLM-5.2 and 5.3 are both very solid. And with the Kimi base model, they know the recipe, and they are turning the crank on the thing right now: scale RL environments, scale RL, figure out the right user distribution, make it longer-horizon. They can do this and will keep doing well at it. If OpenAI and Anthropic find a new paradigm or really change something, then I don’t know. But in the near term, they will just keep charging ahead, and the pace of progress won’t be that different.

Jasmine Sun: People are saying that because the Chinese labs partly rely on distillation to improve their models, if the US labs stop releasing models, or release them much slower, will Chinese progress slow because they can’t distill? And separately, what if we crack down on distillation more?

Nathan Lambert: I think it takes a lot longer for that to kick in. Anthropic did have this very transparent misuse report, which is so weird with the extent that the Chinese companies rely on proxies and route their customers to Claude. I’m going to need somebody to explain to me why all the Chinese LLM companies are doing that so much. Yes, user prompts are helpful for training, but it seems way bigger than the distillation thing. There’s this huge mass use of the models.

Jasmine Sun: That’s pretty crazy.

Jordan Schneider: And it’s expensive, too. What percentage of these prompts are actually going to be useful to train on? I can’t imagine a super high percentage, versus the ones where you’re focused on extracting reasoning. It’s so funny: the PLA guy designing some undersea drone, asking DeepSeek for help, and it ends up going through Claude. Credit to the MSS. They were posting blog posts about this back in April and May: “Guys, you gotta be careful about this. Don’t give your military secrets to Claude.” But presumably you’d think it would be okay to give them to DeepSeek.

Watching the Chinese security state respond to all of this is going to be fascinating. The example they put in that report is everyone’s biggest fear: leakage of state military secrets to the Americans — not even to the Americans, to this enemy AI company that Dario, sworn enemy of China, has set up.

For years I was reading all of these debates — it’s a very small community of people in the West who follow Chinese AI safety debates — and for a while it was clear you could write it down to zero, because this was not something the government really cared about. All of a sudden Xi gives this speech, the head of the MSS writes a blog post; they’re all starting to take this stuff really seriously. Combine that with the labs starting to do tiered releases: GLM-5.3 was the first time where they said, “This company is going to get this model first,” and released it in a staged way. I can only assume that’s going to become more prevalent as the Chinese government processes both the Hugging Face risk stuff and the leakage that’s clearly inherent in “we’re training off all of this distillation.” This is not actually answering your question, Jasmine, of whether they’ll slow down —

Jasmine Sun: No, but it’s useful.

Jordan Schneider: — but I think the “fuck it, we’re shipping” paradigm China has been in is going to come under increasing pressure, from state security people who are worried about things as well as from the more straightforward AI risk stuff, which you’re already seeing manifest.

Nathan Lambert: At least the open-weight part of the Chinese models has slowed down. They’ve started releasing their models in their own offerings and APIs, and then the weights come two to three weeks later. I wouldn’t say I have a conspiracy theory, but is that partially some internal regulatory thing? All these AI labs in China talk to the government and register their releases. Is that built-in time for more testing and more safety before you upload the weights and they’re everywhere? I wouldn’t be super surprised. I was kind of confused, because Kimi and Z.ai both announced the model and then, almost exactly three weeks later, uploaded the weights to Hugging Face. That’s too weird of a coincidence, because Kimi benefits from uploading the weights if they’re planning on it. It gets the distribution. There’s not a lot of history in the industry of the people building the models giving a ton of extended time to release partners. It’s more, “Here’s the model, hot potato, we’re going to release it. Good luck with your implementation for day-zero readiness.” I think they’d still do that if they could. I don’t know how we could ever verify this hunch, but it seems like a logical response: give us a few weeks before uploading the weights.

Jordan Schneider: Speaking of our jury-pool selection — once we have our super easy, one-page US–China AI treaty, who from the Chinese government are OpenAI and Anthropic going to be cool with giving Slack access to?

Nathan Lambert: I’m going to be an embedded evaluator at Anthropic, OpenAI, Zhipu, MiniMax, and Moonshot. I’m going to split my time between Beijing and San Francisco.

Jordan Schneider: We’re sending you to Beijing, Nathan. It’d be a fun job. You get some good duck out of it.

Nathan Lambert: It would be interesting. It’s an important job. That’s why it’s so sad that the question is, who is going to do this? If the industry commits to this, it’s really important that it works and isn’t just papering over a problem.

The Sacks Clap-Back

Jordan Schneider: Let’s respond to David Sacks. Why isn’t this just a normal business thing? You don’t want to ship something to your customers that you can’t be sure won’t blow up their servers. If this is just a liability question, why do we need to make such a big hubbub about it?

Nathan Lambert: The AI companies know themselves, there’s so much money to be made, and they know that they’re not trusted. So they’re trying to have a fig leaf of somebody who is high-trust to help verify their claims and hold them back from the gigantic piles of revenue they could make if they just raced forward. I don’t know the answer. I’m just spitballing.

Jordan Schneider: Is the gigantic pile of revenue there if you implode Morgan Stanley from the inside, and then all of a sudden it’s, whose fault is it? Dario Amodei’s, or Sam Altman’s?

Nathan Lambert: I have sympathy for the response. I’d like to reread what David Sacks said, because I suspect it comes off a lot more harshly than how you phrased it.

Jordan Schneider: We’re doing the generous reading here.

Nathan Lambert: The generous reading is fair. That’s why some people are saying we should be further investigating and punishing the things that have happened, and looking for more things that have happened that I don’t think have been documented.

Jasmine Sun: Why isn’t there a criminal investigation of the Hugging Face thing? I could imagine that being a reasonable thing to do. This is a white-collar crime. You shouldn’t do that, and OpenAI should pay a fine for having done it. I guess they just settled it between OpenAI and Hugging Face —

Nathan Lambert: I don’t think they settled it. Hugging Face could probably still press charges.

Jasmine Sun: I guess they informally settled it. My assumption was a handshake: OpenAI gives Hugging Face some money.

Nathan Lambert: Waiting for the Nvidia acquisition to close, and then Nvidia sues OpenAI.

Jordan Schneider: Part of the settlement: no more Jalapeño. You’re done making your own chips.

Nathan Lambert: Part of the settlement might have been, “Look, we’re getting acquired, and we don’t want to open our acquirer up to this can of worms.” That’s not good for the industry if that turned out to be the quid pro quo. Or it wouldn’t be a quid pro quo; it’d just be Hugging Face seeing these two discussions going on at once, and they’re the only people who see both.

Jordan Schneider: When companies do something wrong, they hire the law firm that has the former attorney general. Or you have the Warren Commission after the JFK assassination. So who’s the Supreme Court justice we’re bringing in here? Who’d be the dream get to head up METR Trust 2.0?

Nathan Lambert: I feel like Paul Christiano would have been pretty good, but now he’s on the OpenAI board. There are people like that.

Jordan Schneider: Well, Trump said no. If Trump doesn’t say yes to that person, the answer is no.

Jasmine Sun: Jordan, what’s your David Sacks clap-back?

Jordan Schneider: There’s another reading of the David Sacks tweet, which is: okay, I guess now he’s fine with AI slowing down, which in and of itself is a discourse win. That wing used to say safety is a fake thing. Now they’re saying, “Fine, just do it yourselves. Don’t bother me with it.”

Nathan Lambert: There are big shifts like that happening. That’s probably the biggest change.

Jasmine Sun: Dean Ball used to be a liability guy. Dean Ball used to say, “Yeah, just do liability,” right?

Nathan Lambert: Now Dean Ball’s an agent swarm guy.

Jordan Schneider: I do want to pour one out for Jordan’s vibe-coding projects, and science or whatever. There’s got to be some cost to the slowdown. We get these world-changing capabilities a little slower, growth is less. It’s kind of annoying.

Nathan Lambert: It’ll be easier to see the cost when you look back in forty years. These types of technological revolutions take decades, and I think AI will happen on the fast side. But what happens if at year four we cure some major disease, and we had this whole eighteen-month pause where we didn’t know what we were doing? I think that’ll be really clear, and there will be some big upsides. Did we pause and nothing happened? Or did we pause and there were still disasters? There are a lot of ways this plays out. If we do the pause and there are still cyber disasters — say a small city loses power for a week and thousands of people die during the pause — then if you zoom out, it’s pause, minor disaster, quiet period, positive big breakthrough. I think people looking back would be fairly sympathetic, but also potentially frustrated if the breakthrough is big enough. We’ve been doing this for three years, and it’s going to take at least twenty to get the key benefits.

Jordan Schneider: Dario and Sam used to keep saying, “As we push the frontier, we get better at safety, because we learn the questions that need to be solved.” Now everyone’s saying there’s an alignment overhang — we could have everyone do research on this stuff and get better at it.

Jasmine Sun: One argument is that if they slow down frontier training, they’ll spend more effort on diffusion and inference right now, and maybe that’s good and we get more of the benefits of AI. But if we’re worried about how fast we get the cancer cures, there’s got to be some cost. The counter would be that doing a bunch of bio RL to get the cancer cures will also enable these things to build bioweapons, and so we’re willing to slow down both of those things.

Nathan Lambert: I don’t think pacing is going to solve the human competition factor, and I think the human competition factor is the primary source of unsafety. They want to win the bag, and you’re going to really have to pause things for the bag to no longer be on the table. Right now it’s making the smaller models — the models we have — way cheaper to serve. And once you do your next RL run, if your model is ten times as fast, you RL ten times as hard. It’s really hard to know what exactly happens.

Jordan Schneider: So you think during the pause we’ll get better AI writing? Will that be okay to improve?

Jasmine Sun: I can’t imagine. You could argue super-persuasion will be improved, so we’re not allowed to improve writing capabilities. But there’s something nice here. I was talking to somebody at one of the labs yesterday, and right now everything that’s not on the path to RSI gets neglected, including writing, but also a bunch of other useful stuff. If we get a bunch of real-world, mundane-utility progress because we stopped focusing on RSI, that might be kind of great. We get good applications, we get good writing models. There’s a lot of stuff that’s not RSI-useful but useful to normal people. If there’s a way to pace frontier development while accelerating the mundane utility applications and diffusion stuff, that would be cool. But I’m not technically expert enough to know whether you can do that in a way that makes sense.

Nathan Lambert: I think you can. The models are smarter in the sense that their capabilities are higher, but that’s from these RL environments, and a lot of the RL environments are kind of broken and the models are kind of cooked, where they forget stupid things. It could take time. I call it “rounding off” the models: a good model is a good run, the little pieces played well together, it has nice soft smooth edges, you know how to work with it, and the benchmark scores feel real. I feel like a lot of the models during this RL-RSI race are honestly tuned for internal deployments. Somebody in my blog’s Discord said, “GPT-6 — that model is designed to be used in agent swarms. Us plebs running it one at a time is kind of weird.” Maybe true. Why do so many people think Claude Opus 5 is bad? I bet Anthropic gets a lot of value out of it.

Jasmine Sun: The Claude models in particular get more incoherent from a natural-language perspective. They have more annoying personalities; they talk to you in stranger and more alien ways, because they’re not trying to be helpful to normal people doing Google lookups or writing emails anymore. They’re willing to accept some worse emails to get better RSI. If we slow down the frontier stuff and improve the other stuff, that would be cool.

Nathan Lambert: That’s why we have Gemini.

Jasmine Sun: Has anyone checked in on Gemini lately?

Jordan Schneider: Oh my god. Demis. Demis is going to come back from the grave. Could Demis run our eval thing? That’d be such a loss in status, though. That’d be really sad.

Nathan Lambert: Gemini’s doing okay. That sounds wonderful. Demis gets promoted out.

Jasmine Sun: Send Demis. He just has a desk at OpenAI, but with a different color on the badge, so you know.

Jordan Schneider: He gets a little badge. The scarlet D.

Jasmine Sun: This short little British guy buzzing around the office. I like that idea. I think it’s a good idea.

Nathan Lambert: I think it’s one of the best ideas we’ve had. Ilya — SSI can be an independent evaluator.

Jasmine Sun: Oh my gosh. Bring Ilya back to OpenAI, but as an independent evaluator.

Nathan Lambert: Like an intern.

China’s Not Pacing

Nathan Lambert: Question back on the content. Say Kimi K3.5 releases and it’s another leading Chinese model. Is that going to trigger some weird discourse — “China’s not pacing the frontier”? It seems like the knee-jerk reaction is going to be “China’s not pacing, we are, China bad.” What’s going to happen in the next few weeks?

Jordan Schneider: I think pacing is going to happen on the margins, if it happens at all. Until you have a real government regulator that can say thumbs up or thumbs down, it’ll happen on the margins, and probably won’t, if we’re talking in months. How much of that quote-unquote lead is going to be quote-unquote cut by whatever soft norms get created by some independent evaluator over the next six months? Really not a lot, I don’t think.

Nathan Lambert: What happened when OpenAI paused their frontier RL runs for a few weeks? What did they spend the resources on?

Jordan Schneider: What did people do? They still went to work, right?

Nathan Lambert: They still made the model smarter. They just didn’t do the big run, I’d guess — the big uncertain thing.

Jasmine Sun: Testing robustness. I’m sure they found some stuff to do. But it’s a good question, what happens if a really good Chinese model comes out, or Meta releases a really good model.

Nathan Lambert: Meta said they’re going to open-weight theirs.

Jasmine Sun: If they release a frontier model, OpenAI and Anthropic will stop pacing, right?

Nathan Lambert: Or are they going to go full-court regulatory press on Meta?

Jordan Schneider: That’s the other thing: the Trump administration has a say. If Trump says, “Zuck, no,” or “Zuck, you have to pace” —

Jasmine Sun: But does Trump seem like he’s about to tell Zuck to pace? I just don’t think that’s where the admin’s going, explicitly. Trump and Zuck are buddies.

Nathan Lambert: No, he doesn’t. That’s important — this Trump post that we started with.

Jordan Schneider: Again, two things are true at the same time. They stopped Fable because they were scared of it, right? And they want to beat China.

Jasmine Sun: And because they don’t like Anthropic. Do you not get a world where Trump says, “Yeah, Mark, he’s my buddy. He’s been donating to me, he’s aligned, we’re happy to let Meta have their frontier model. Anthropic wants to pace? Okay, have fun, doomers.”

Jordan Schneider: I think this is a salient enough political thing that we’ve crossed some threshold. There are futures where it does not matter that your CEO is into MMA and has the fucking UFC president on the board. If there is national outcry and segments on Fox & Friends saying this is a crazy, dangerous, horrible thing that’s going to kill us all, I don’t think the president says, “But Mark Zuckerberg knows jiu-jitsu.”

Jasmine Sun: I agree, and that doesn’t seem to be what Trump is doing right now. But maybe his Truth Social posts are just stupid rhetoric we shouldn’t pay attention to.

Jordan Schneider: Look, we have peace with Iran, we’re at war with Iran. All I’m saying is we can swing. Yes, on September 14, 2026, this seems implausible, but I could see it going in a different direction.

Nathan Lambert: That seems right. This could just be Trump being like, “The SF power circle got too much attention this weekend. They’re not right.”

Jordan Schneider: “You think these guys are good? No, come on. What are you talking about?”

Jasmine Sun: Cool. Back to Trump.

Jordan Schneider: We can’t end with him. Okay: how do we get the most benefit out of this? How is GDP the highest twenty years from now? That’s what I want. There’s some equation with a number of variables. The safety things are good enough that you can deploy the models. You keep getting enough progress that the models get better and deliver you productivity growth. And you don’t have enough hiccups along the way that the government freaks out, or the people freak out and say, “We don’t want this anymore,” and it gets shut down like nuclear.

Nathan Lambert: It felt like the frontier labs were self-regulating enough on that already. The intellectual capture of them as safety orgs is pretty strong, at least at Anthropic. I don’t know about OpenAI, but the average populace there cares about this so much. And I have pretty high faith in human ingenuity to solve problems, so it seemed like the maximize-GDP route without extinction was the one we were on. The problem is if Anthropic drinks the Kool-Aid a bit too much — “RSI equals extinction” — which didn’t seem like the regime we were in, to me. That’s why it’s a hard needle to thread.

Lu Xun Maxing

Jordan Schneider: Let’s close on something else. I’ve been Lu Xun–maxing of late. Dario writes this article about pacing the frontier, and I truly believe this is Dario being Dario. There’s not a ton of filter; it’s just him telling the world what he thinks. So much of what I consume for my day job covering China is not that, because there are constraints on how people write, and they need to write within a system. I would love to read Liang Wenfeng’s 2,000-word essay responding to pacing the frontier. We’re not going to get that, because it is not the prerogative of CEOs or commentators or thinkers to be aggressive and thoughtful and provocative in public when you’re talking about things that are ultimately within the government’s purview.

So something in me said, let’s read some great Chinese books. I finished Rulin Waishi 儒林外史 (The Scholars), a satire of Qing dynasty officialdom, and Fortress Besieged 围城, a Republican-era satire: marriage drama, people coming back from studying in France who think they’re too cool to hang out with the Chinese hoi polloi. But Lu Xun 鲁迅, who I’ve not spent enough time with — I’ve been working my way through Dawn Blossoms Plucked at Dusk 朝花夕拾, an essay-collection memoir of sorts about his early life and upbringing. He gives this critique of old China. He’s talking about the death of his dad, and going to all these traditional Chinese medicine people who are buying beaten drum skin, because that will beat the disease out of his dad. On the one hand, he clearly thinks this is absurd. But he also has this eye of empathy and sympathy: okay, if this is the entire world you live in, you are giving people some comfort and solace in this ancient worldview, which is now collapsing under the influence of Western thought and science. He’s not bullshitting you at all. A lot of the Chinese writing I read is people bullshitting themselves, or giving you 30% of it. It’s a real breath of fresh air. Lu Xun is super accessible; good translations are out there, and he writes episodically, so you can read five pages before bed every night. It’s a much better way to China-max than watching TikToks of Chongqing.

Jasmine Sun: Maybe we’ll get Jordan’s “What Chinese Literature Taught Me About DeepSeek’s Pacing Strategy.”

Jordan Schneider: The Chinese name for DeepSeek is Shendu Qiusuo 深度求索. It comes from a poem by Qu Yuan 屈原, which is basically about political disappointment and moral persistence. He’s a poet-official whose rivals kicked him out of the government. The ruler didn’t want him, and he’s hanging out in exile, trying to do the right thing. It’s a classic Chinese trope: the leader didn’t listen to me, so I’m going to be upright and moral even outside the system. In China today, you don’t get to be outside. The way you get outside the system is you move to the Bay Area, if you want to exist as a clean atom, not touched by anything. I think it’s telling that someone like Liang Wenfeng — who clearly has that independence of thought, the Lu Xun–ish person who is not bullshitting you whenever he talks — at some level understands this. I don’t know if it’s leading him to ruin, but it puts him on a different plane than a lot of the other people who have made peace in a more direct way with the role that they play with the state. Sorry, that was much longer than I thought I was going to go.

Jasmine Sun: Poetic. Good place to end.

To receive new posts and support our work, subscribe!

Mood Music from ChinaTalk Records