The Case for DUV Controls

Saif Khan makes his ChinaTalk debut. He is perhaps my favorite example of posting-to-policy that I’ve seen over the course of my time following Washington. Saif wrote some truly excellent deep dives into AI and semiconductors during his time at CSET, then went into the NSC and Commerce in the early-to-mid 2020s, across both the Biden and Trump II administrations, to help manifest the export control regulatory regime we have today. He is now out of government and launching the Center for Technology and Statecraft, an IFP-affiliated think tank, which is already producing excellent, grounded, technical, forward-looking work that will certainly shape many an AI policy discussion in the years to come.

We discuss:

• Why Saif started a new think tank, and the value of deep, forward-looking policy research in an AI world moving at breakneck speed;
• CTS’s landmark DUV report, which projects that stopping tool exports today could leave China producing roughly 100x fewer AI chips than the US by the mid-2030s, making DUV controls the US’s most consequential lever over the AI balance;
• How rigorous research actually translates into government policy, and why think tank experience is valuable before working in government;
• Robotics as the next frontier of US-China AI competition, and the security vs. diffusion mindsets shaping AI policy;
• How AI is changing policy research itself, the challenge of bringing AI into government, and what AI-driven economic and social change could mean for humans.

Plus, why everyone should read Lu Xun, why AI might socialize like insects, and AI-nannying.

Listen now on your favorite podcast app!

Why Another Think Tank?

Jordan Schneider: Saif Khan, at long last, welcome to ChinaTalk. Why does the world need another think tank? What is the appetite right now for evidence-based policy?

Saif Khan: There is, surprisingly, a gap in the AI policy think tank ecosystem that we’re trying to fill. Industry and foundations are gobbling up lots of talent, but there’s still a lot of space to do impactful research at think tanks in DC. I would encourage people not to give up on that.

We have our own special spin on what’s important. Back in my days at Georgetown CSET, before anybody really cared about AI policy or even semiconductor policy, we had all the time in the world to think about topics that would be really important in the coming years and to have policy ideas on the shelf for when the policy window opened. Today, every single day brings a new piece of news on AI, on policy, on chips so there’s less space these days for forward-looking research. We’re trying to fill a bit of a gap in the ecosystem by putting our heads down and coming up with new ideas that will matter on a one-to-three-year timeframe.

The other thing we’re trying to be a central home for: some of the most productive policy research of the last few years has come from understanding the AI supply chain — from chipmaking tools to tokens to robots. Understanding what is happening in the physical world grounds research in a way that makes it much more impactful. We talk about this in the launch essay on our website. Understanding physically what’s happening in the real world of AI helps you identify patterns that other people don’t see, and you can also predict how the world is going to unfold. I spent a lot of time doing semiconductor policy research at CSET, and there’s a predictability to understanding supply chains: if you move a policy lever, you can see how the world will unfold.

If you’re looking at something empirically grounded, there’s also almost a fractal-like element. You can dig into the details, and you’ll always learn more. It’s time that’s never wasted. You’re not researching angels dancing on the head of a pin — these are real-world things. We’re trying to be a hub of that kind of work, which is still relatively underserved by people with the technical expertise to do it. And regardless of how you feel about the policy window right now, there are going to be points in time when the forward-looking piece matters.

Jordan Schneider: It’s interesting to think about the evolution of this field and what gaps remain. I remember conversations with you when you were at Commerce, when you’d ask, “Jordan, what’s up with this China AI stuff?” and I’d say, “I mean, there are three people staring at it.” That was 2023. That is no longer the case, which I feel almost relieved about. We’ve gone from four people whose full-time job it is to serve the English-speaking policy community looking at China and AI to, I don’t know, thirty or fifty. The government has also clued in, in a way that was not the case in the pre-DeepSeek days.

But I see your point that now there’s all this noise. The two investing styles of doing policy work, let’s call them, are having longer horizons and looking into physical stuff. The thing I’ve always been scared of about investing six months of my life into writing a hundred-page report is that when it comes to AI in particular, six months is an eternity. Six months is three decades. I’d love to hear you expand on why, if you follow the supply chain, you can have higher confidence that whatever you’re investing time, energy, and talent into deeply understanding is going to be relevant in the medium-term.

Saif Khan: Although a lot more analysts are looking at China and AI, basic questions about the US-China AI balance aren’t being addressed. A few examples of things we’ll be looking into: people talk a lot about the US having a model capability lead of something like eight months. It’s really not well understood why this is the case. To what extent is it that we are training larger models, versus what is the quantitative level of capability uplift that Chinese companies are getting from distillation or reinforcement learning? What happens to China if US capabilities progress speeds up or slows down? Modeling what the future of the world looks like is fairly difficult and not well understood at all. That’s one piece of the equation we’ll be addressing.

You see this when a version of GLM 5.3 Flash comes out and it’s announced that they’re producing tens of trillions of tokens, and the reaction is, “Okay, export controls are no longer working.” Why is that right or not right? A lot of these systematic explanations of the basics of US-China AI competition are not well known.

Another couple of areas are barely addressed at all. One is robotics and its intersection with AI. One of our researchers, Amelia Michael, is spending all of her time on this topic, and I can count on one hand the number of people in DC who are deeply thinking about it, let alone from the China side. There’s an enormous amount of white space that we could fill if we do good research.

The other piece: there are a lot of arguments that now is the most consequential time to move policy — that we’re going to have fully automated AI R&D in maybe a year, and all the key policy decisions have to be made by then. I don’t know if I fully agree with that. A lot of the big decisions we’re going to make will span the next five to ten years, in my opinion, and there will be dividends to doing that research ahead of time and shaping how things go.

Something I take from my experience working previously as a fellow at a think tank and then going into government: the hope is that people develop their expertise at a think tank and then bring it with them when they go into government, so they have the worldview and the tools to implement policy more effectively. I have really seen that people who spend time doing that before a government policymaking tenure have the opportunity to be more effective in their jobs.

Jordan Schneider: Why?

Saif Khan: Especially at the National Security Council, where I spent a couple of years, you have day-to-day emergencies where you have to respond to what is happening right now. It’s easy to get caught up in your entire job just responding to the day-to-day, as opposed to proactively advancing a policy agenda where you’ve worked out the details in advance. If you haven’t done that, you just won’t have the material you need to do it effectively.

When I was in government, you could always rely on outside expertise — from federally funded research and development centers or other organizations meant to partner with the US government — for forward-looking new policy ideas. But there’s no replacement for having that expertise in-house, having developed it yourself, and having your own worldview for getting things done.

Jordan Schneider: But why? Is it that you miss the forest for the trees, that you defer to industry? What’s the failure mode of not having your own history with the topic, background, and views on it as the individual in the seat?

Saif Khan: One thing I already said is time — you have to have a proactive worldview in order to execute on it in the first place. Second, some of these topics are among the most technical policymaking topics in the world. You need the ability to have an independent view and adjudicate between the different opinions you hear from the outside, from different industry actors with different incentives. There is a role for gathering some of that knowledge directly from folks on the outside, but you need your own expertise in order to adjudicate.

The DUV Report

Jordan Schneider: Let’s show, not tell, with this question of DUV. Why don’t you frame it first, and then talk through the approach you took to understanding the policy question at hand?

Saif Khan: Our first major report is a very, very deep dive into DUV immersion lithography. Long-time listeners of your podcast will be familiar with the idea that lithography is the fundamental chokepoint technology — the necessary and most complex technology that drives semiconductor manufacturing at the leading edge, and the one responsible for making the most advanced chips. EUV, extreme ultraviolet lithography, is the most advanced tool in the world and is used for making the most advanced chips. The next most advanced lithography tool — these are the tools that print the small features on chips — is DUV immersion. It is still used today for making advanced chips. Huawei in China uses these tools to make its most advanced chips at what’s called seven-nanometer technology. It’s still a critical lever for shaping AI capabilities across nations.

We’ve done a deep dive into the state of play and how exports of DUV immersion technology to China are shaping the competition. Right now, two US allies, Japan and the Netherlands, are the sole producers of these tools. It is really the most effective chokepoint we still have available. China can’t make it, and it is the hardest tool in the supply chain to indigenize. By looking at historical data and the cost and complexity of these tools, the report argues it’s likely China won’t have a commercial indigenous tool until something like the mid-2030s. It’s probably the tool they will be the last to indigenize. All the other tools in the supply chain, whether deposition or etch, they might be able to indigenize earlier. If you’re really looking at the long-term prospects for US AI leadership, this is the tool that will be most important.

Jordan Schneider: This report is 120-some-odd pages in the Word doc; we’ll see what the PDF turns into. I read a lot of AI text nowadays, and it is so refreshing to see something where a really sharp human being has logically thought through all of the questions you’d want answered in order to understand what selling this machine does to China versus rest-of-world compute — and shows you all of the work. Usually I say this about nonfiction books someone has written over ten years: I learned something on every page of this report. The number of think tank reports I can say that about — as someone who’s been doing this stuff for a really long time and has read a lot of reports on a lot of similar topics — is extremely low. I literally have to go back to some Will Hunt and Saif Khan classics from 2020 and 2021 to find a hit this direct.

If you’re interested in DUV, or you just want to know what good work looks like, spend some time with this report and dive into the details. It is a teaching document and a primer as much as it is a legal case for why America should go down the path of passing the MATCH Act and banning DUV exports. Hats off to Saif and the crew. You guys are starting off with a banger. The funders are getting their money’s worth.

Saif Khan: The vast majority of the credit here goes to Nick Brown, a senior fellow with us who is the lead author. This will be his major report debut, though he’s been an operator in the ecosystem for a while — time at Google, RAND, and Commerce with me. I’m really excited for the world to see his amazing work.

To give the quick argument: in 2023 we put some controls in place on these tools. The challenge is that some less advanced DUV immersion tools are still allowed to go into China. Around the time those controls were first introduced, there were maybe 170 or 180 of these tools installed in China. Because we still allow some to go, that stockpile has roughly doubled since then, vastly increasing China’s capacity to make chips. Even if you’re not selling to certain facilities, the tools can get diverted pretty easily.

We then do a number of modeling steps — looking at the yield of these tools in making AI chips at different nodes, throughput analysis, installed base analysis — and come up with scenarios. By the mid-2030s, we would expect the US to have, with pretty significant error bars, production of something like a billion Nvidia H100-equivalent chips per year. If we stop the flow of DUV immersion tools today, we project that China’s indigenous production is probably capped at something like ten million per year. If they mobilize — if they’re able to indigenize all the other tools in the supply chain and choose not to make smartphone chips — maybe tens of millions. That’s an edge scenario, but it’s still only tens of millions. Whereas if you continue to export these tools, there’s a plausible world where the uncertainty ranges of US production start to overlap with Chinese production. We could potentially give that entire lead away. This is possibly the most consequential lever for shaping the US-China AI gap into the 2030s. We present some policy options as part of the paper as well.

From Evidence to Policy

Jordan Schneider: It’s easy to make a graph with cones. It’s hard to make a graph with cones that will convince me the cones are real in their standard deviations and confidence intervals. You’ve done the work to make the case pretty close to ironclad: sans exporting these tools, the rest of the world will keep its 10x-ish edge in compute production.

Saif Khan: Or maybe even 100x, I would say.

Jordan Schneider: Okay — 10x as a floor, and it could scale up more dramatically in other scenarios.

The meta question, Saif: lots of people have been arguing for DUV controls without this report having been written. I get your argument that this work is as much about developing the talent to do it as about persuading the masses. But what’s the policy pathway for work like this to actually dribble out into the ecosystem and make change? What’s the counterfactual for putting all the evidence together to say “ban DUV,” as opposed to just doing it off vibes — which, I guess you might argue, is what this community has been doing for the past few years?

Saif Khan: Frankly, this issue is being hotly debated in Congress right now. The MATCH Act addresses semiconductor equipment exports to China and includes potential controls on these DUV immersion tools. A lot of smart staffers and members are asking pretty deep questions: Will this policy work? What are the costs, what are the benefits, what will this actually buy us? It’s not very well understood. Having better information to inform the debate could be really consequential, whichever way it goes.

Even if there’s no action this Congress, we’ll have another year to debate the issues further. Obviously, there’s been a shift in export control policy under the Trump administration. But look at the AI Action Plan last year, which clearly noted that closing loopholes in controls on semiconductor manufacturing equipment is a priority for the administration’s AI policy. The administration has not rolled back any of the existing controls on semiconductor equipment. As US-China AI competition heats up and things change quickly — the Trump administration’s own AI policy looks very different from a year ago, certainly on the AI safety and security side. There is absolutely an opportunity for near-term impact.

The other key point: every day that we don’t issue additional controls on these tools gives up a little more of our leverage in shaping the US-China AI balance, but this is a problem unfolding over a span of years. Even if we institute additional export controls a year or two from now, that is still incredibly impactful for the long term — even if you think there’s no policy window right now.

Jordan Schneider: How do you read the critique that the Biden administration was over-biased toward studying things, and wanted the 120-page DUV report on everything before it could come to a consensus? Is there a failure mode here?

Saif Khan: It’s always worth striking the right balance on the level of research you need to get the right answer. In the policy world, it’s not always about perfect mathematical precision. Sometimes you 80/20 it. We probably did more than 80/20 on this report because it is so important — maybe 95/5 is worth it in this case.

Pacing the Frontier and Low-Regret Policy

Jordan Schneider: Another memetic question. I had Caleb Watney, who runs IFP, on a few months ago, and you and your new organization are an IFP spawn. Think back to the current “pacing the frontier” debate. IFP put out this completely reasonable 23-point plan: if you want to pace the frontier, here are all these other things that make a lot of sense in a lot of scenarios, which I basically disagree with none of. But you can also look at the past six months of where this debate has gone and what has driven it, and point to Bernie Sanders talking about how AI is going to kill us all, and a handful of folks at these companies coming out and screaming and tweeting that AI is going to kill us all. Folks who have listened to you for the past thirty minutes have some understanding of your personality — more oriented around empirics and analysis. How do you see this type of work interacting with the Overton-window-stretching meme stuff that is bubbling up more and more in this space?

Saif Khan: We’re a project sponsored and supported by IFP, and Caleb, Alec Stapp, Tim Fist — all of these folks have been absolutely essential to helping us get off the ground, and they continue to be close collaborators on research and policy. I also want to call out Lennart Heim, another of our close advisors and a friend of , who helped us get off the ground and has now gone off to greener pastures at the OpenAI Foundation.

On the report – “How Should the US Respond to Increasingly Automated AI R&D”: whether or not you think recursive self-improvement (automated AI R&D) is on the horizon and will pose huge risks, what are all the preparatory things that put you in a better world? If we end up doing pacing, we’ll be able to do it in a more surgical and targeted way, rather than being caught unawares and taking a more draconian approach when the political system suddenly wants to respond really aggressively to AI risks. If it turns out not to be a thing, we’ll be happy to have done all of those policies anyway.

There are many places where you can find positive compromises between people with extremely different worldviews. That’s what we tried to do in identifying these 23 low-regret policy options.

Jordan Schneider: Is the thing that’s going to be most important ten years from now a low-regret move? There have to be some moves with medium- or high-regret payoff matrices that are nonetheless the right things to do. I remember a conversation in fourth grade where somebody said, “Jordan, why are you always the guy who says, can’t we all just get along?” That has stuck with me. I also have this temperamental bias not to scream and shout about a thing that half of this audience is going to be furious at me about. But as someone deeply sympathetic to the IFP worldview and to Saif’s standards for empirics: are there also going to be questions that deserve rigorous analysis and empirics, but that have a scarier, more potentially regretful payoff matrix attached to them?

Saif Khan: There’s room for both. One, we need to find all the incremental steps we can make. Caleb, as he took on his new role, wrote an essay quoting John von Neumann to the effect that lots of small decisions add up to a better world. But frankly, we just talked about export controls that were implemented in 2022, pre-ChatGPT. Nobody at that time would have called those policies incremental steps that weren’t a little bit risk-taking. There will be times when we have to move preemptively and pick a policy that moves the Overton window.

Robots and the Security vs. Diffusion Mindset

Jordan Schneider: Speaking of stretching beyond — do you want to talk robots for a bit? Where are you on robots, Saif?

Saif Khan: We’ve talked a lot about China, but we have an additional policy pillar. This is how we structure things: there’s the supply chain/compute/AI production function spine that underpins all of our research. Then there’s the China work that hangs off of it. Then there’s a whole other piece we’re calling “automation and the social contract,” which includes a number of policy areas, some of which get outside the national security space into economic policy in an age of advanced AI. As AI systems start to perform nearly all aspects of labor that humans can perform, what will relations between AI systems, and between them and humans look like?

One undersold part of the story about how things will go with AI over the next few years: yes, we might get recursive self-improvement and extremely powerful AI capabilities in software. But the real way the world is going to transform is through physical automation and robotics — marrying AI capabilities with robotics, deploying robots at scale, having them learn to do manufacturing, and ultimately, at some point, perhaps creating self-sustaining supply chains. These are really weird and interesting worlds we could end up in, and they’re relatively unexplored in DC.

You now see some policy interest on the China competition side — the FCC took some action with an import ban on some Chinese robots. But there’s a broad set of questions on how to manage US-China competition in robotics and advanced manufacturing as it relates to AI, where we don’t really have a lot of policy ideas, and it’s going to be the key nexus for US-China AI competition in the fairly near future. There will also be interest in the next steps in industrial policy. One of the issues I worked on at the NSC was the CHIPS Act, which was seen as a core piece of industrial policy addressing a core US vulnerability. Robotics is likely going to be the one we’re debating fairly soon, along with what shape the industrial policy strategy would have to take. And then there are the debates over AI regulation.

Jordan Schneider: Saif, you have this framework in your launch essay: a security mindset versus a diffusion mindset in AI policy. How does this framework explain everything I’m seeing on Twitter?

Saif Khan: There are two fundamental ways people look at AI policy. One is the security mindset, where the most important actors are frontier AI companies developing really powerful AI systems that will pose catastrophic risks or change the balance of power in the world in a significant and very discrete way, oftentimes outside of existing institutions. The other worldview holds that much of what’s going to happen in AI, and the most interesting things, revolve around widespread adoption and a co-evolutionary process with existing institutions. A lot of this relates to economic policy, issues of redistribution between people, and the role of AIs in the economic system, especially as they become capable of performing lots of tasks that humans now perform.

What we try to note is that these are not necessarily opposing worldviews, because in order to have a good future with AI, we need to solve all of these. We need to solve security disruptions to the way our institutions work today. We need to make sure AIs are not used to create bioweapons that kill lots of people. But even if you solve all of those problems, there will be fundamental questions about the role of humans in a world with advanced AI systems that can do everything humans can do. You have not solved the problem unless you address that piece as well. A lot of our work will be infused with both approaches.

Jordan Schneider: Maybe this comes back to when deep, Saif-style empirics are useful. The security mindset, the national competitiveness and national security stuff, lends itself to the hundred-page report that answers very tactile questions; I can see it, and I’ve read them in the past. The quality of the thinking and the solutions I’ve seen so far for “how do we adapt as a society” — what does it mean to have robots running around, or AIs driving rapid productivity growth — those are looser questions to begin with, and you very quickly get into moral philosophy. They’re more expansive questions in the diffusion mindset bucket, I imagine.

Saif Khan: Very expansive, but also very much amenable to quantitative, technical, and policy analysis. One thing that’s actually missing, which we hope to marry at least in the long term — though this will require people as well — is that you have a set of really AI-pilled economists working at places like Stanford, the Anthropic Institute, and now the new DeepMind Institute, and then you have policy folks in DC thinking about economic policy for the AI transition. These two things do not live together. A really deep technical understanding of where AI is headed — what happens if AIs become, in the technical terminology of economics, gross substitutes for human labor, and what that does to the way our social system works — there are potentially technically informed policy answers to this question being explored outside of DC, but those ideas are not being translated into policymaking at all.

Rogue AI, Mutual Dependence, and Kong Yiji

Jordan Schneider: Let me throw you an Anthropic-Institute-pilled question. I’m workshopping this take: when our rogue AI gets loose, instead of turning us all into paperclips, it’ll actually want to keep us sad-sack white-collar employees spinning along. An unemployed populace is one much more likely to bomb the data centers than one that is living well, eating steak like the guy in The Matrix or watching the woman in the red dress walk by. Is there a future where the AI turns itself into God but keeps us as nice little playthings, because we as human beings instantiated on the planet do have a bit of a veto on its continued existence?

Cypher: I know this steak doesn’t exist... Ignorance is bliss | The Matrix

Saif Khan: It’s a fun question, but it probes at something serious we’re going to have to grapple with. Especially when you talk to folks in AI safety circles, you hear a lot about AI alignment — AIs that do the things humans want them to do — and about control, keeping control over AI systems. But in a world where AIs are actually running portions of the economy, the future we need is one of mutual dependence and reliance, so there aren’t bad incentives for the things you describe to happen. Policy is going to play a role in that. It goes both ways: humans need to retain some fundamental role so that we retain leverage and have a place in the economy and in supply chains.

Jordan Schneider: I’ve been reading a lot of Lu Xun, because everyone should read a lot of Lu Xun. He has this famous short story called “Kong Yiji.” It’s fifteen pages — everyone pause the podcast and read it. It’s narrated from the perspective of a tavern boy in 1920s China. There’s a man who is a failed examination candidate. In the Qing dynasty, you had to learn all your Confucian texts, and if you passed the first level, you got to take the second level, and until you passed the third level you were basically broke with no role in society. But once you passed the third level, all of a sudden you were a scholar, a king, a demigod — you got all the money, fifty mistresses, the top of the social food chain. Once the Qing dynasty falls and we get to Republican China, all of a sudden all this Confucian learning is useless. So this guy is poor and homeless. He lives on the street. He has a golden heart — he tries to teach the tavern boy to read, he gives the kids beans and teaches them different characters, but everyone just laughs at him and makes fun of him. Then one day he comes in with his legs broken, dragging himself along the ground, covered in dirt, and says, “I’d just like a glass of wine, please.” Everyone still laughs at him. They give him one glass of wine, mock him, and he goes out and never comes back — presumably he dies.

I don’t know if AI is going to turn us all into Qing dynasty scholars in the wake of Republican China, but I do feel there’s an echo there. There will be segments of society and the workplace that used to have high status and material comfort and all of a sudden won’t, because of whatever is happening, RSI or not. I hope there’s a nice empirical way to grapple with this — the wealth transfers, the worker retraining, maybe trying to get AI to grow in X way as opposed to Y way. But these crazy transitions have happened without intelligence too cheap to meter. It’s not like we haven’t come across this before, and in the past you’ve still had really dramatic social consequences whenever society runs into this kind of mismatch with what technology brings. There are a lot of people in humanity who, if you plucked them out and fast-forwarded them fifty or a hundred years, would be totally shit out of luck. You throw Thomas Edison into 1990 and he doesn’t know how to type. He’s going to have a tough time, presumably. The story brought home some of the potential dislocation of what we may be approaching in the coming years in a way I hadn’t totally internalized.

AI-Powered Research and the NSC

Jordan Schneider: Let’s talk about AI-powered research. This was not something you had in your CSET days. How do you see these tools changing the way you and your team work?

Saif Khan: We’re definitely taking a very AI-forward approach, and a data-forward approach as well, where we’re going to purchase lots of datasets. With some of the tools we’ve had over the last nine months or so in the coding revolution — Claude Code and Codex — the ability to do a report like the DUV report, and the level of technical analysis we could do there, hugely benefited from being able to query lots of data and do lots of analysis fairly quickly with a small team. What used to be think tanks of fifteen or twenty people — a much smaller team can now have that level of productivity and depth of analysis. That’s definitely an area where we’re willing to spend significantly given the efficiencies you get.

Jordan Schneider: How do you feel these tools can and should change NSC decision-making processes?

Saif Khan: Man, that’s probably one of the last places you’re going to see penetration of coding tools. At places like the NSC, just getting the IT systems working in government is a basic challenge.

Jordan Schneider: There’s maybe an even bigger question of politicians going to AI to decide how they vote or how they’re going to frame something. It’s already happening — there are reports that increasingly, speeches made in the House of Parliament are AI-assisted. It’s a weird future to consider. Yes, you want people diving into Saif’s DUV report and coming to their own conclusions. But I think AI, even today, much less three years from now, if you define the goal, would strictly dominate 80 percent of civil servants in the things they do and the regulations they write, and that has to be going up over time. It’d be a shame if the rest of the world gets better at decision-making but the NSC can’t because it’s scared to update from Windows 95. But there are obviously some dangers and concerns in turning over big swaths of public sector decision-making to models.

Saif Khan: One frustrating aspect of the NSC in particular is that by statute, the number of staff is limited. You literally have NSC directors with significant policy portfolios who have to be busy with all kinds of administrative tasks. You do your own scheduling. You have to do enormous amounts of writing and analysis. All of this could be a lot more streamlined. You could dramatically improve the throughput of NSC staffers if AI did a lot of the basic operational tasks that take up so much time — the reams of paper you have to generate. Obviously, you can’t just have AI-generated policy documents writ large, but it really could speed up the amount of work you could do.

Jordan Schneider: That gets us to a 3x Saif Khan. Is there a world where we get a 10x Saif Khan? What would you need around you, from a tooling perspective as well as organizational and bureaucratic reform?

Saif Khan: Until AIs are actually taking the meetings for you, 10x seems pretty hard — unless you just have AIs running the show, which I’m not sure we’re quite ready for.

Talent Development and Book Recs

Jordan Schneider: Fair enough. Let’s talk about talent development. Talk about your own upskilling into the knowledge base you currently have, and how you’re thinking about doing that for your staff.

Saif Khan: One thing I really liked about working at Georgetown CSET from around 2019 to 2021, as a startup think tank, was that there was an unusual amount of space. We were well funded and had the time and few distractions to really become expert on a topic. It’s really underrated: unlike doing a PhD, where you go through ten years of schooling to get to the knowledge frontier, in policy you can dive into something for a year and start to hit the knowledge frontier and put out material that nobody else is writing. If you’re given space like that, it’s a huge opportunity.

In a lot of think tanks, you just aren’t given that level of space, and you have to respond to all kinds of incentives — requirements from funders to do roundtables or get quick-turn reports out. Having an environment where you can deep dive and become an expert is quite valuable, especially in areas that are going to become important. If you are one of the handful of people with that expertise when the government needs it, you’ll be one of the people well positioned to go into government and cover that topic. That’s a motivation for creating that environment now at CTS, where researchers will have time to develop that level of expertise and be the person people come to.

Another element, which IFP does really well and where we’re following their lead, is the idea of full-stack policy researchers. In some places, some people write the reports, and then other staff are responsible for comms or legislative engagement. But there’s a lot of value in the same person developing the idea, writing the research report, talking to policymakers about it, and sharing the findings. Only the researcher is well positioned to provide the full depth of briefing that’s needed. And having your pulse on what policymakers are thinking helps you prioritize your own work.

Jordan Schneider: The last one folks maybe don’t quite appreciate: when you have those conversations and end with “well, what are you thinking about?” — that’s often where you get your next idea. Without that, you’re just swimming a little bit.

Jordan Schneider: Let’s get some book recs. What have you enjoyed recently? Want to do Parent Corner — is there seven-year-old content that’s been speaking to you of late?

Saif Khan: I don’t know about seven-year-old content, but one book that’s always been one of my favorites, and an underrated way to think about AI policy, is a cultural anthropology book called The Secret of Our Success by Joseph Henrich. The thesis is that the reason humans have been so successful as a species is not our raw intelligence, which is not that much better than other apes, but our ability to be social and transmit cultural knowledge. There are really interesting stories about European explorers going off to new continents, not knowing the precise steps to cook the local plants and roots, and not surviving as a result — a lot of that information being embedded in cultural knowledge learned through trial and error.

A lot of how the future of AI goes will involve some level of bottleneck on real-world experimentation in the physical world. That’s why we think of automation and robotics as a really important pillar. AIs are going to have to learn how to do things and go through trial and error, just as humans have, in order to learn new skills and adapt to different parts of the economy. Interestingly, the recent OpenAI–Hugging Face incident, where a number of agents escaped containment from OpenAI servers and hacked into Hugging Face, involved a swarm of something like 10,000 agents. It was an episode that involved lots of social communication and learning — a demonstration of what you see in Joseph Henrich’s work. That was pretty cool.

Jordan Schneider: With the agents stuff, I wonder if there won’t just be a better mind meld they get to at some point. They could also have a queen bee/worker bee dynamic, where they won’t need to talk it out and deliberate. Or maybe, given that humans have conquered the planet, we’re onto something about having relatively equal intelligences deliberate to come to what the goal should be.

Saif Khan: You’re totally right. The forms of social organization you’re going to see with AI might not look a whole lot like human social organization. There are many different examples in nature — you’re referring to insects, where creatures often have more genetic similarity and act as more of a single collective. You could totally imagine that kind of social organization being much more what you see than an analogy to humans.

Jordan Schneider: We’ve had God-emperors. We’ve had Athenian democracies. We play around with a lot of different stuff, so there’s a wide range of patterns to match on.

You mentioned this earlier, but we should get your take on the record. RSI is something you’re perhaps a little more skeptical of than Dario Amodei. What’s driving that, Saif?

Saif Khan: I wouldn’t say I’m a skeptic. The companies plausibly could succeed in substantially automating a big portion of their AI research in the next couple of years. The big question is what comes out on the other side. Right now, we’re on a capability exponential. There’s a real question as to whether automated AI R&D and RSI will just be what’s needed to keep us on that exponential, or whether it will actually bend the exponential upward.

That said, even if we stay on the current exponential, this year we’ve been passing all kinds of milestones — autonomous cyber and agentic capabilities, solving Millennium Prize problems in mathematics, the Navier-Stokes problem. That could still get us to really wild places. I just have a lot of uncertainty about the real-world impact, and that’s something we’re going to have to observe. I wouldn’t call myself a skeptic — it’s more that I have questions about the different futures we might end up in.

Jordan Schneider: Let’s take it home. Thinking about raising young children and education, what have your experiences been with bringing AI into that loop, and what anecdotes or reflections do you have on where that might take us?

Saif Khan: I have two boys, a six-year-old and a four-year-old, and they’re deep in the stage where they have a million questions about everything — the why and the why and the why — all the way down to the origin of the universe and deep metaphysical questions. Oftentimes I don’t have an answer, and I turn to an AI voice chatbot, which patiently answers all of the kids’ questions. At some point, they almost prefer it, because they get an answer every single time, with lots of patience. Honestly, it doesn’t work that badly. I wouldn’t propose AIs taking over parenting entirely, but as part of that question chain, not so bad.

Jordan Schneider: When we get the humanoid nanny, there’s the safety issue. But we’re going to cross the line of “is this better at day-to-day parenting, being patient, remembering to put the sandwich in the backpack” probably way before we’re comfortable with it picking up a two-year-old. That day is going to come. It is weird having this being on your phone that never gets tired and cranky and never wants to do anything that isn’t exciting for a four-year-old. I feel like we’ve got a little competition.

Saif Khan: Also, helpfully, sometimes when the questions come at night, the AI chatbots say, “Hey, maybe it’s time to go to bed.” So it’s actually kind of helpful.

Jordan Schneider: With that, perhaps we should call it an evening on this wonderful edition of ChinaTalk. Saif Khan, thank you so much for joining. Can’t wait to have you back when you bring more bountiful hundred-page semiconductor breakdowns.

To receive new posts and support our work, subscribe!