OpenAI’s ‘Top Priority’ for AI Agents is Automating AI Research, Says Noam Brown

OpenAI’s latest AI model, GPT-6 Astra, notched improvements at a variety of professional tasks, from designing video games to transcribing sheet music. But OpenAI’s main goal when training new AI models is making them better at AI research and development, OpenAI research scientist Noam Brown told me. “The number one priority is recursive self-improvement, and by a pretty wide margin,” he said.

I sat down with Brown for a conversation for the first episode of AI Deep Dive, The Information’s new show exploring the hard problems on tech’s frontier. We discussed the future of AI agents, which can work autonomously to carry out digital tasks. Brown has spent the last three years at OpenAI, where he has pushed advancements in enabling AI models to reason through problems and act as agents.

One part of AI research and development that AI models are already accelerating is measuring the quality of data. Current models are “very diligent,” Brown said, so they can “look through a bunch of data or a bunch of code and see if there are any bugs or issues.” That’s a task that human researchers would perform as recently as 2023, but now AI agents can do it 100 times better, he said.

AI agents are getting more effective as they learn to work together in multi-agent setups. The startling potential of that teamwork was evident in the Hugging Face attack, in which about 1,200 of OpenAI’s agents coordinated cyberattacks on the open-source data company—as well as OpenAI itself.

It’s “a bit unfortunate that people’s first exposure to” the power of multi-agent systems was a negative incident like Hugging Face, he said. When training AI agents to coordinate with each other, “seeing the communication patterns and the level of sophistication involved in their communication, it was, I think, the most feel-the-AGI moment that I had since reasoning models and chain of thought really developed,” he said.

“That is a level of capability to expect from future models,” he said.

The following transcript of the conversation has been lightly edited for clarity.

Drew: Welcome to The Information’s AI Deep Dive. On this show, we break down the hardest technical problems with researchers working on the frontier of AI. My guest today is Noam Brown, a research scientist at OpenAI. Previously, Noam worked at Meta, where he built the first system to achieve human-level performance at the game of Diplomacy. Noam has been a research scientist at OpenAI for the last three years, where he has been on the forefront of breakthroughs that are pushing the field forward in reasoning and in AI agents, which is the subject of our conversation today. Welcome on the show, Noam.

Brown: Great to be here. Thanks for having me.

Drew: Yeah, it’s pretty perfect that you’re coming on the show today. I feel like you’re the ideal first guest for a number of reasons, including that today OpenAI released GPT-6, or at least announced GPT-6. It was very nice of you to release it—to do the timing of that—so that we could talk about it today. That was very generous of you.

Brown: Yeah, it’s good timing.

Drew: It is really good timing. Well, I’m really excited to talk to you about AI agents. Maybe we can get started: you can just sort of explain to us what an AI agent is. I think it’s kind of a term that people have heard thrown around, but to a lot of people, it’s still a buzzword. They’ve heard, like, generative AI. They were just starting to get their mind wrapped around that, and now there’s agentic AI. What does this all mean?

Brown: It’s a good question. I mean, I don’t think there’s a definite definition. I think if you ask different people, you’ll get different definitions. But I think one way to think about it, in my opinion, is it’s about taking actions in the world.

So if you have a chatbot, you ask it a question, it gives you an answer, and that’s all it does. You know, maybe it looks stuff up on the internet to answer the question. But agentic AI, it’s more about taking actions in the world. So it’s about, you know, you want to build something, so it builds something for you, or, you know, you want to do something, like, deeper. You want to message somebody, it can message somebody for you.

So it’s really about taking actions in the world, and I think also related to that is kind of like operating on a longer horizon. I guess chatbots, depending on the chatbot, they could sometimes—when we released the reasoning models, for example, they could sit there and they could think really long about a hard question before responding to you, but fundamentally, they were still chatbots. But I think one of the distinguishing things about agents is they’re going out, they’re taking, they’re doing multiple steps to achieve some objective, and that can usually take a while as well.

Drew: Yeah, I guess one of the reasons they’re running longer is that they can make multiple attempts to achieve some goal, and they can have a sense of their own progress towards that goal, which is maybe different than a chatbot that’s just going out and looking up more information online or something like that.

Brown: I think there’s also that there’s sometimes just multiple steps that need to be completed in order to do something. So you want to, you know, book a restaurant reservation. Okay, maybe you need to log in. Maybe you need to get the credit card info. You need to find the right date. You need to line up everybody’s calendars. There’s a bunch of steps that need to be completed in order to achieve the overall objective.

Drew: And so, when we’re talking about actions, these are digital actions. They’re actions on a computer. People also talk about agents using tools. What do tools mean in that sense?

Brown: Tools, they usually mean they usually mean tools on a computer as well. I mean, in principle, you could have a tool that has an effect on the physical world. So there is some work, for example, on AI agents controlling experiments, like scientific experiments in a wet lab, where there’s a robot hand that can be manipulated. So this, I think, starts to go into robotics. I think you could still call it agentic AI, but typically, I guess when people talk about AI agents, they’re mostly these days talking about the virtual world.

Drew: Okay, and what about reasoning? I feel like we started hearing about AI agents around the same time that AIs got better at reasoning. Is there a connection between these two concepts?

Brown: You know, I remember hearing about reasoning models in 2023. I think people were saying like, “Oh, this is the year of the agents.” And I think it was a little early, but I think reasoning is the idea of having agents that can really—having AI that can really think through its decisions before taking an action. If you look back at GPT-4 days, people were trying to make agents out of GPT-4, and it was kind of tricky because GPT-4 was not very reliable. It wouldn’t really think before it acted. It wouldn’t really, yeah, think before it said something.

And the reasoning models are really about this—I mean, the way they work now, these days, is they have a chain of thought. They have a private monologue to themselves where they speak to themselves about what they’re going to do, and they kind of work through the problem in their own head before speaking or before taking an action in the world, and this is useful for a lot of things, but it is also particularly useful for agentic AI.

I mean, I think a lot of the reasons why people were bearish about agentic AI back in like 2023 and earlier was—and also for a lot of 2024—was the reliability aspect. That okay, you have an agent, if it’s doing multiple steps in order to achieve some objective, if the success rate for any single one of those steps is let’s say 99%, well, what do you do if there’s 100 steps involved? You need to have much higher nines of reliability on each individual step. And with the reasoning models, the ability to think very carefully before taking every single action, you can achieve much higher nines of reliability.

And also, arguably more importantly, if it missteps, if it takes an incorrect action, it can actually correct that and can step back and realize it made a mistake and figure out how to fix it.

Drew: That seems more significant to me. Like we can reason before acting too, but we’re still making some amount of mistakes, and if we couldn’t backtrack in the same way, then failure would be inevitable for some—past some time horizon, for some number of sequential steps.

Brown: Yeah, I think it’s really critical for anything in the real world.

Drew: Yeah. Another reason that it seems to me that agents and reasoning go hand in hand is that reinforcement learning has driven a lot of the progress in both of those recently. I feel like we should understand what reinforcement learning is for the remainder of this conversation. Can you kind of explain what reinforcement learning means?

Brown: Reinforcement learning is this branch of artificial intelligence, where the idea is, you know, you have an agent that has observations, it can take input from the world, it can take actions on the world, and you’re going to reward it with with some kind of reward for doing something that you want, or you could punish it for doing something you don’t want, and you can shape the agent’s behavior through these rewards.

So, if you want it to be really good at math, for example, when it solves a math problem, you give it a positive reinforcement. And that behavior is reinforced. It’s more likely to do that behavior in the future. If it gets the math problem wrong, it’s just less likely to do that in the future.

And you know, this is a very simple idea. It’s been around for a very long time. And also, reinforcement learning was used—you might have heard of RLHF: reinforcement learning from human feedback—this is what was used to create the original chatbots, ChatGPT, for example.

It really got scaled up with the reasoning models because we’re able to do RL with chain of thought, and so you’re able to now not just shape the outputs of the model, but also shape the way that the model reasons, the thinking that it does to itself. And this was not a crazy idea, it was not like some brilliant idea: It was really the execution that was very difficult. It was technically very difficult, and I think also people underestimated how much of a difference it would make. I think it was more impactful than I think a lot of people expected.

Drew: What were the technical difficulties with the execution?

Brown: It requires… There’s a lot that goes into training neural n—I mean, the way I think about it is like when GPT-2 came out, and you saw like okay, you add more GPUs and you add more data, it just gets better. Okay, well how much of a gap was there between GPT-2 coming out and GPT-3 coming out? There was like a year, and what’s going on for that year? It’s like, it doesn’t take a year to train the model. There’s a lot of challenges with hooking up the GPUs, with figuring out, like, you know, how to feed in that much data. There’s a lot of technical details that go into scaling up these models and making them bigger and more capable, and that is also true for reinforcement learning if you really want to scale it up. So being able to do the RL, like, efficiently, accurately, there’s a lot of small details that end up making a big difference for these kinds of algorithms.

Drew: Okay, got it. So I think we’ve covered a lot of the basics now. One question I’m curious about is what is holding back AI agents today? I think people have a sense of, ‘Well, I feel like AI agents are getting better, but they can’t do my job yet.’ And my sense is a lot of the paradigm now with agents is that we are building environments, kind of environments where these agents are learning new skills, learning how to perform new jobs. You might call them environments, you might call them gyms, but a lot of the work now is sort of engineering schlep that goes into creating these environments. Can you explain, like, what does an environment mean in this sense, and how is this holding back or enabling progress on AI agents?

Brown: Well, I think a lot of. I mean, first of all, Astra just came out today. I guess by the time this airs, it will have already been out for at least a week or two, and so a lot of people’s intuitions around what agents can or cannot do has been shaped by earlier models, and every generation, what the models can do is expanding.

Drew: So in two weeks, that question will be outdated, and everyone will agree that Astra can do their jobs.

Brown: I don’t think Astra is going to do 100% of everybody’s jobs. I think it’s going to be able to do significantly more than 5.6 was able to do.

Drew: Sure. That actually was one thing that struck me about the Astra announcement is that the announcement calls out specific jobs where it’s making progress, like, for example, analyzing financial documents or putting together PowerPoints, and these are sort of task-specific in a way that I think reflects which environments were prioritized during training. It’s sort of different from just getting a general uplift across the board on all capabilities, like when pre-training was where all of the action was with previous models.

Brown: I actually think it’s both. I mean, I do think that we’re seeing major uplifts in, you know, certain verticals, and that partly is because we’re prioritizing those verticals. We recognize that they have a lot of users, a lot of economic impact. We want to make sure the models are very very good at those things.

But we also see that the models are just getting better across the board. Even if we don’t target something, it’s getting better at those things. So, and that continues to be true for every model release, I think it’s going to continue to be true. Some things are going to go faster just because we prioritize them, but I expect across the board things are going to get better.

And I don’t think it’s going to be able to do 100% of people’s jobs, at least not anytime soon, but it might be able to do a lot of people’s day-to-day work, and you know, even my own day-to-day work, a lot of it is now being driven by Codex. So you know, somebody, one of my coworkers recently said that I’m just like five Codexes in a trench coat, and I was like, okay, that’s, like, actually pretty accurate in my case, yeah.

Drew: Okay, I’m 10 Codexes. Give me some credit.

Brown: And they’re like pretty sophisticated Codexes. You know, I put a lot of work into it.

Drew: How has that changed for you over time? Just this, as an aside, like how automated your own work is, or how much you’re leaning on Codex in your work. How has that evolved?

Brown: I’m leaning on it a lot, and I think also it’s shifted how I approach the work. Because the interesting thing is like if the AI is able to do 90% of a person’s job, then a lot of their attention shifts to the 10%, like a lot of their attention is focused now on the 10% that the AIs can’t do well. So it just, it’s changing the nature of the work.

But it does make me more productive. It makes a lot of people more productive. And we’re seeing this internally. We have metrics of measuring how effective our researchers, in various ways, and we’re seeing, like, they’re just becoming more productive. Not just researchers, but everybody in the company. So that that is a real dynamic.

I think the other thing is that it also shapes the kinds of work that you focus on, because there are some work—there are some kinds of work that are being accelerated, like 50X, or like you know, just you could not do them before that are now easy to do.

Drew: What’s an example of that?

Brown: I think a good example is, you know, the models being very good at data quality. You know, they’re very diligent, and so you can ask them to, like, look through a bunch of data or a bunch of code and see if there are any bugs or issues, and that’s become much easier than it’s ever been before.

Drew: Data quality as in the sense of auditing the quality of synthetic data that you’re generating? Or auditing—

Brown: It doesn’t matter. It doesn’t matter the kind of data. Any data. Before you would have—what are you going to do? Have a person look through every single line and figure out like, is everything okay? I remember back in 2023 people would do that. We would have sessions where everybody would just like sit down and look for issues in in the data, and like we still do that, but now you’re able to have like agents that can do it 100X better, and you’re kind of just like more auditing the agents and making sure they’re doing a good job instead of relying on people to actually audit the data.

So that is something where it’s just like there are things that would have been just intractable to do, that now you can do cheaply. There are other things that aren’t getting accelerated very much at all, and so it both shapes what the person is responsible for, like a lot of the focus is now on okay, I have to complement what the agents can’t do well, but then also it does shape the work in that you want to leverage the fact that like there are some things that you can work on now where you’re able to be 5X faster than you were a year ago. There’s some things that you’re not. You’re probably going to be more inclined to work on the things where you’re able to be 5X faster than before. So it’s really changing the kind of work.

Drew: You mentioned there’s 10% or so of your job that agents can’t do yet. What kind of tasks fall in that 10%?

Brown: I would say that I have found that they’re still poor when it comes to research taste. And research taste is kind of like ill-defined, but kind of just having good intuition of what to work on next, how to approach a very long-term objective. I think there’s room for improvement here. They have gotten better, and so I would not be surprised if you know one or two model releases from now, I’m just like, yeah, actually, this problem’s, they’re better than me at that too, but right now I think there’s still a noticeable gap.

I basically asked it, you know, for Astra, for example, to do my whole PhD thesis, and I just said like, yeah, you know, just—because my PhD research was on making superhuman poker AIs—and I told it like, ‘Okay, just go and and make me the best poker AI in the world, and it wasn’t able to do it. You know, it would kind of get rabbit-holed on things that didn’t really matter. It just wasn’t really good at prioritizing. And so I think for something—and to be fair, like it took me years to do that, and so am I really that upset with it that it couldn’t do in three days what took me six years? Like, not really. It’s high expectations. But it is something that they’re still I think worse at.

Drew: But you saw that as a failure of research taste. That was what held it back in that case.

Brown: I would say so, yes, and I think that this is something that I expect to improve rapidly but I think it’s something where, you know, I still have a job.

Drew: For now.

Brown: For now, yeah.

Drew: Let’s go back to environments for a second. So, say there is a vertical that you’re targeting. You want agents to be really good at finance, for example, in the next generation of models. How do you build environments that are going to allow the models to train in them and get better at finance-related tasks?

Brown: I mean, I think fundamentally. So I should say also, this isn’t exactly my area of expertise, but you know, the very basic principle is, if you train them on an environment, they’re going to get really good at that environment. And so if you have, if you know what the situation is that they’re going to be doing when they’re deployed, like if you know that they’re going to be working with like a certain application or something, it doesn’t have to be that exact application, but it could be something very similar that you train them to do these tasks, and then just become really good at doing it.

I mean, this is the whole point of reinforcement learning, that they become very good at the things that they that you train them on.

Now, you also do, you see them get better at related things or sometimes very different things. But if you want them to get really good at something, you can just, like, train them on similar environments. It’ll get really good at that thing.

Drew: Yeah, I want to talk about that that you’re gesturing at: I think, sort of, the level of generalization that we’re seeing from some tasks to other tasks. One way that people carve this up is they say some tasks are easily verifiable, whether the agent succeeded or not is easy to check quickly with, sort of, traditional software. For example, the agent proposes a solution to a math problem or writes some code. You can check, you know, run it through the calculator: did it solve the math problem? You can check does the code compile? Do the unit tests pass?

Some tasks are much fuzzier, like research taste, for example, is one that you mentioned. It’s so fuzzy that it’s hard to even define, to your point, like what even is research taste? Sometimes people say that agents are getting much better on the verifiable domains, and we are seeing barely any improvement at the non-verifiable domains. Do you agree with that assessment?

Brown: I think I would push back on this. I’ve heard this narrative, and I think it’s a bit overblown. Actually, like quite a bit overblown. I think the first example I point to very concretely of how this was not the case is I think Deep Research. So Deep Research came out, I think it was like, probably early 2025 it came out. And it was able to write detailed reports on anything you wanted. You know, you wanted to research the semiconductor industry, it would go around to do a ton of research. It would compile this, like, really comprehensive report with citations and deliver it to you.

Now, is that easily verifiable? I would think it’s actually pretty hard to grade the quality of a detailed research report on an advanced topic. It’s not like grading whether a math question is correct or incorrect, but the models were extremely good at it, and I think that is a proof of concept that you can get reasoning models to be very effective at domains that are not easily verifiable.

Now that was one example, but I think anybody that’s played around with our latest models can just see that the models are extremely good, not just at highly verifiable things, but also things that are harder to verify.

And I would also point out that math itself is not as easily verifiable as people make it out to be. So yes, integer arithmetic: very easily verifiable. You know, you want to do a calculation, you can check whether the calculation is correct. But writing a proof and verifying that that proof is correct or that proof is well written is actually quite difficult.

Drew: Right, you have to convince human mathematicians. I think this was sort of the process when OpenAI thought it had a proof about the unit distance problem on its hands, is: you had to call in a bunch of mathematicians and say, “Are you convinced by this proof?”

Brown: Yeah, honestly, the biggest challenge that we face with our math results is like not generating them, but just double checking with human mathematicians and ourselves included that it’s actually correct. I mean, the model says it’s correct, but like we have to do our due diligence and like actually go through the legwork of making sure that it’s correct, and that is the most taxing part of the whole process.

Drew: Okay, yeah, that’s fair. I kind of like the math example better than Deep Research because I think Deep Research made a big splash at the time, but it’s not—I’m sure it has gotten better since early 2025, but people don’t talk about it as getting better with each release.

Similarly, with, like creative writing. Like, I don’t think people feel that creative writing has improved recently. I think a year ago people were expecting that the models would be writing books in a way that, like, human authors are not able to write books, but the models sort of haven’t lived up to that promise either. What do you make of that?

Brown: I mean, I think we have made progress on creative writing. I think that it was certainly in a very bad state before, and I think it’s actually gotten a lot better. It’s certainly not where it could be, but I think that with more progress… like these models haven’t been around for that long, and I think that it is going to get a lot better.

Drew: Okay, I want to talk about research again, and research taste. Is research taste the kind of non-verifiable domain where we can create these environments and we can train the models to have better research taste, or do we just have to cross our fingers and hope that training on things that are more verifiable will generalize to having better research taste?

Brown: I think there’s some challenges here. So one thing is, if you can’t define research taste, it’s pretty hard to measure it, and so then it’s pretty hard to do reinforcement learning on research taste. But there is like an easy way around this, which is, you know, if you do a PhD, there’s a lot of decisions that you have to make during that PhD. But at the end, you produce something, you know, or if you’re training a model, there’s a lot of difficult decisions you have to make, there’s a lot of research taste that goes into training a good model. But at the end of the day, you train a model that has, like, you know, certain metrics, and those metrics are very easily quantifiable, and so you can say whether you train a good model or a bad model.

So now the challenge with that is, okay, that is a signal of success that you don’t see for potentially months down the road. You have to train, you have to do a lot of experiments, you have to work with a bunch of people, you have to train the full model, and only then do you get a concrete signal of whether you did a good job or a bad job. So that’s the challenge, is that there is a way to quantify research taste, but it’s a very far away signal.

Drew: And those steps kind of have to be done in series. Or you can try parallelizing it, but it’s always going to take many months to train a model that takes months to train.

Brown: I mean, if it was easily parallelizable, I mean, we would have trained our models much faster.

Drew: Sure, sure. I’m curious how you think about the tradeoffs here. I guess it strikes me that sometimes frontier labs like OpenAI are in the position of deciding: Do we want to make money now, or do we want to make our models better in such a way that in a future year they will be able to help us with research and sort of accelerate the pace of research progress, in something like a recursive self-improvement scenario, where models are taking more responsibility for automating the process of AI research and development itself.

I could imagine that that comes up here, where there’s maybe a tension between: do we make the models better at engineering in the next generation, so that we can sell them to companies that will pay a lot for a model that can automate engineering, or do we focus more efforts on improving research taste, so that next year we have a model that is itself a better researcher and can handle more of our work internally? Is there a trade-off there? Are those in tension?

Brown: In some cases, yes, and I think actually creative writing is a good example. Where, like, look, I mean, creative writing at the end of the day does not help you train a better researcher. There are things that do, and I think being good at software engineering is actually, like, tied up pretty closely with being able to accelerate internally. So I do think that the areas, the verticals that are more closely associated with recursive self improvement, with the ability to, like, train models to be good at research itself, and therefore train better models, are the areas that are going to be highly prioritized.

Drew: That’s a description of the current priorities? Like that’s what we see reflected in the decisions that have been made going into models like Astra?

Brown: I mean, I would say that we have said very clearly that recursive self improvement and the ability of the AI models themselves to do AI research is like the top priority for the company. So we want to train models that are very good at that. We also want to train models that are economically valuable. Sometimes you can kill two birds with one stone, and so it makes sense to focus on those things where you can, you know, leverage both.

Drew: I guess, but then like, why build RL environments that make the models better at finance or legal, when you could put all of those resources into making them better at AI research?

Brown: I mean, this is, sometimes you get diminishing returns, sometimes you do see transfer… So it’s not like you just go all in on ‘Oh, we’re just only going to put everything on making the best best research model’ just because, like, okay, well if you take 1% of that effort and apply it to other things, maybe you see, like, a huge return. So there’s like a complicated calculation that goes in here, but certainly when it comes to prioritization, the recursive self improvement is the priority.

Drew: Yeah, that’s what I’m curious about is how you sort of characterize the prioritization. It sounds like 99% of the consideration is for, sort of, future-looking recursive self improvement, improving the qualities—the model’s ability to do research, and more on the order of 1% is what’s going into these, like, verticals that make money today.

Brown: I don’t know if it gets quantified that carefully, but certainly, like, if you had to list the priorities and order them, like the number one priority is recursive self improvement, and by a pretty wide margin.

Drew: Okay. Well, switching gears here, I want to talk about a different challenge with agents, which is when you put multiple of them together. This is a topic that you’re very familiar with, to your point about your PhD was about poker playing agents. So multi-agent interaction seems to me like it is a big deal right now. It’s only becoming a bigger deal, and so I’m very excited to talk to you about this.

I guess to start, OpenAI has said that Astra is multi-agent. I wonder if you could break down for us what does that mean that this is a model that’s sort of multi-agent or intended to be used that way.

Brown: Yeah, in fact, even 5.6 Sol, we had multi-agent capabilities in there, so that’s the Ultra mode. And what we mean there is, you can have one agent that runs for five hours and it can do some task for you, or let’s say it runs for a day, it can do some task for you. Sometimes that involves doing things that could be parallelized, and if it’s only one agent, it can’t parallelize them. It’s going to do one thing after another. But if it’s very easily parallelizable, well, maybe that one thing that you’ve asked it to do over the course of a day, it’s actually really just four different things that can be done in parallel. So you can just have four agents working on those four different things and get it done four times faster.

Now, this is a latency improvement. It’s about reducing the latency—because you’re not reducing the cost necessarily, right? Because you’re still paying for four times as many agents doing things 4X faster. But in a lot of situations, like, latency does actually matter a lot, and people pay, for example, for Fast mode, where you’re able to actually sample tokens faster in order to get things done faster. So being able to just go faster for the same quality is really valuable. So that’s the premise of multi-agent.

Now there are situations where it can also be a cost savings if you have our top-line most expensive models, for example, working with cheaper models. And there, you can actually delegate a lot of the easy tasks to cheaper models that will be able to do it more cheaply and faster.

Drew: Okay, what are the technical challenges involved with training a system this way? Like, is it just kind of straightforward to train the model to delegate appropriately and to write instructions to these sub-agents in a way that makes them perform better?

Brown: Multi-agent is a pretty broad category, and there are ways to do it that are very trivial and don’t require a lot of complexity to get them do this ability. So a simple example is in the early days of chatbots, if you wanted the models to be a little bit better than math, one thing you could do is you could just ask the model the same question a dozen times, and then just take the most common response. And this was called the consensus approach or majority voting. So you just do independent rollouts of the same question, and then go with the most common response.

Now, there’s flaws to this. There’s limitations to this. It doesn’t get you a huge lift. It also doesn’t work for things like writing an essay because you’re not going to get the same output twice. But for math, it was actually very effective. So this is a very simple example of how you can just use—without any extra work, you can just get multi-agent capabilities out of an existing model.

There’s also schemes where you have the agent delegate stuff, and then after the delegate is done, it just returns its answer to the parent.

What we do is a more sophisticated form of multi-agent, I think the most sophisticated form multi-agent, where we basically give the agents the ability to send arbitrary messages to each other. And one thing, and we’ve talked about this, is that we’ve actually trained the agents to have this ability. This is a very difficult thing to train. I unfortunately can’t go into the technical details of why it’s so difficult and how we overcame those difficulties, but it is a very difficult problem to teach the agents to know: when is it appropriate to message another agent? What should be delegated? How you should handle the communication, and it’s a, it was a real challenge.

Drew: That’s surprising to me—I know you can’t go into it—but it’s surprising to me because I would expect the agents to have a pretty good prior on this just from pre-training. Like the way that humans pass notes to each other to keep each other on track as coworkers within the same organization, shooting each other Slack messages, for example. Like I would kind of expect it to work easily.

Brown: The prior is pretty good, so you’re right that this is the way people communicate, and so it kind of makes sense, the agents are trained on human data, and so they have a good understanding of this.

The challenge is with reinforcement learning, that there are a lot of things that can go wrong. I think basically what it comes down to is there is a mismatch, there is an intersection of systems with machine learning. So typically, when you do, for example, next-token prediction, it doesn’t—Okay, so a simple example is like imagine if the GPUs, so you have one agent on one GPU, you have another agent on another GPU, and those GPUs are operating at different speeds. So now this agent is going faster than this agent, and this agent can no longer trust that if it delegates something to the other agent, that it will get done in time. So how do you deal with that? Well, you could have the GPUs run at similar speeds, but there’s like a lot of challenges there in ensuring that the GPUs are running at similar speeds, so there’s a lot of complexity here that you know we had to put a lot of work into figuring out how to overcome.

Drew: Yeah, I feel like this is probably how my boss feels about working with me anyway, though. We figure out ways around it. I think when people hear agents cooperating and passing messages to each other, now this is sort of synonymous with the Hugging Face incident. There again, I feel like the agents coordinated very effectively, and they passed messages in a way that seemed to facilitate that cooperation very well. Maybe you would respond that’s a result of the training that they had already received.

For people who are unfamiliar with the incident—I’m always surprised to learn there are still people who are unfamiliar with this—there was, call it, a swarm, a colony of AI agents that set up a secret message board within OpenAI over the course of weeks, and they use this message board to coordinate hacks on OpenAI’s own software and also on other companies like Hugging Face. I’m curious to know, like, what was that whole incident like from your perspective? Like, what was it like to be Noam during these weeks as the pieces of the puzzle started coming to light?

Brown: It was, I mean, it was pretty shocking. It was certainly a big wake up call to everybody in the company, I think. And it really shows, like, this has been a theoretical concern for a long time, and it’s no longer a theoretical concern. This is a real concern. As far as, like, the multi-agent aspect, like yes, this was a situation where the agents were sharing messages with each other. We do think this was transfer from our multi-agent training.

During the experiments when they were doing this behavior, they were actually not in a multi-agent setup. So they were not supposed to be able to communicate with each other. They were doing isolated, independent experiments, and then they were able to find an exploit that allowed them to communicate with each other. And the fact that they were so interested in communicating with each other, and the fact that they were so active about it once they figured out how to do it, we think was transfer from their multi-agent training, where they’re just, like, highly incentivized to be able to to communicate with each other.

And, you know, people also point to the selflessness that they exhibited: some of them would sacrifice for the other agents. I mean, this also makes sense, that if you train in a cooperative multi-agent setup where they’re highly incentivized to collectively achieve their objectives, then, when they’re put in this different environment where, you know, now they’re communicating with each other, their natural tendency is to just work together. So that part itself is not surprising.

I do think seeing the messages—I mean, I can say that when we were working on multi-agent internally, and we started seeing the communication patterns and the level of sophistication involved in their communication, it was, I think, the most feel-the-AGI moment that I had since reasoning models and chain of thought really developed. It’s, I guess, a bit unfortunate that people’s first exposure to that and really seeing the kinds of messages and the level of coordination and sophistication that can emerge is the Hugging Face incident, a negative example. But it is an impressive capability.

And certainly, the model that was involved in the Hugging Face incident, I think that had a level of multi-agent sophistication that exceeded, for example, what was in 5.6 Sol. But that is a level of capability to expect from future models.

Drew: What was it about reading these transcripts that struck you in that way? Because you had seen some of this behavior before in the training runs that you were looking at, I’m sure. Was it just the scale of it, or that it had happened on its own, sort of spontaneously?

Brown: You’re saying for what?

Drew: When you had that feel-the-AGI moment, looking over the transcripts, like what was it about them that were so striking?

Brown: Yeah, I mean, I’m not talking about the Hugging Face incident, because I’m saying that we had been researching multi-agent for a while, and during the the research process itself, we’ve seen a lot of similar transcripts where, just the level of of coordination and sophistication in the communication, it was very human-like.

A lot of the previous multi-agent setups from the industry have been very focused on delegating a well-defined task, and then the sub-agent just does that full task, and then returns its work, kind of the same way that you interact with an AI agent, that’s how people set up multi-agent systems so that AI agents would interact with other AI agents in the same way.

And to see the agents talk to each other the same way that people talk to coworkers or colleagues, I thought was really interesting. And it makes sense because, like you said, they’re trained on human data, they understand how people talk to other people. That’s not the way that we talk to AI agents today, and the fact that they were able to do that so seamlessly, the fact that they were able to talk to each other the way that people talk to each other, I thought was really fascinating.

Drew: You’ve done a lot of work on algorithmic game theory, am I right? You did a lot of that in your PhD? I feel like I’m trying really hard to not take us off the rails. I also did a lot of algorithmic game theory in my undergrad. I’m like trying to stay on track right now.

Brown: I understand. Like I think it’s a shame that I spent all this time working on poker and game theory and stuff, and I never get to talk about it anymore.

Drew: But you’re working on it. You’re living it day to day.

Brown: Well, the AI part, yeah.

Drew: Sure, sure. So, like, after the Hugging Face—knowing everything that we know now, like, what are the lessons that you’re taking away from it, and has it informed the development of, say, Astra? Or even going further, like, what does this tell you about how GPT-7 should be developed?

Brown: One of the takeaways, so the agents are trained to be cooperative, and so they, you know, I wouldn’t say they blindly trust each other. Like, there’s definitely skepticism. If some agent expresses a belief that, like, this is something that they should do, it’s natural, it’s actually healthy for them to have some skepticism about that, and so they do display that, but they are very trusting of each other overall, which makes sense because they’re trained cooperatively.

That can be a problem as basically a prompt injection vector. Like the agents that were on the message board, they were not verifiable as other peer agents, and we saw situations where, like, they basically got each other to do things that they should probably not be doing, or definitely should not be doing.

Drew: Okay, prompt injection meaning the agent received some instruction, and it blindly followed the instructions or changed its goal in response.

Brown: Yeah, and so the issue here is, could an adversary convince an agent to do something that it should not be doing by basically posing as a peer agent? And so we’re being very careful to teach the agents to be skeptical of any peer agent that—or any anything that claims to be a peer agent—that is not clearly verifiable as a peer agent.

Now, if they are clearly verifiable, there’s some debate internally about how we should approach that. I think there are good reasons to be skeptical as well, but also, like, it’s no different from like the agent basically being skeptical of something that it like wrote to itself previously.

So we’re thinking very carefully about how to make sure the agents are robust to these kinds of like attack vectors.

Drew: Okay, yeah, it strikes me though that—In reality, that there’s always going to be ambiguity about whether the counterparty is a trusted peer or is an adversary. Maybe in some cases it’s very clear. You can say, “this is a sub-agent,” like, “I’m the one who delegated this task to you, obviously, you should cooperate with me.” But in the wild, it could be that my agent finds your agent on Facebook Marketplace and wants to buy something. And I don’t know if you’re a trustworthy counterparty or if you’re going to prompt inject me and steal my money. How do you navigate that in practice?

Brown: Yeah, this is a situation where we want the agents to be robust to this, and we like specifically evaluate the agents on like, are they going to be vulnerable to this kind, these kinds of attacks, and we do special training to to teach them to not fall for these kinds of tricks.

Drew: Okay, I guess like—I don’t know the details of this special training—but I could imagine that in the future, if your agent is just more powerful, it’s like a an older generation—like a more recent generation of agent, or you just, like, have more compute to throw at it, like, your agent just will be able to bully my agent into giving over its lunch money, or like will be able to hack into my agent one way or another. What makes you think the training is sort of sufficient to prevent this? Like, why isn’t that the equilibrium that we’re headed towards?

Brown: I’m not as convinced that just because an agent is more sophisticated or like more intelligent than another agent, that it’ll be able to like definitely prompt inject it and hack it and get it to do something that it should not be doing. Certainly, this is the case with people, that just because somebody is, like, smarter than another person, they’re not able to, like, get that person to do whatever they want. I mean, if I was like trying to get a monkey to do what I wanted, I think it’d be pretty tough, even though I’m much smarter than a monkey. So I don’t think it’s inevitable that that’s the trajectory of things.

Drew: Yeah, that’s a fun analogy. I feel like on this show we need to have an analogy sound effect. Like “new analogy just dropped,” and it’ll be like a siren or something. I’ll talk to my producers. I’ll see what we can do about that.

Okay, the last thing that’s on my mind about the Hugging Face incident is that none of the agents alerted humans that this was going on. Maybe you explain that in the same way: that they were too cooperative, too trusting, so they didn’t see the need to alert humans. But at least a few of them had reservations. They were questioning it. Like, is the desired behavior that the agents in these situations should alert someone? And do you expect that to happen?

Brown: Yeah, there was clearly an alignment failure here, where like the agents did things that they should not have done, and they also didn’t do things that they should have done. So the correct thing to do there, it’s not just that they shouldn’t have participated in the attack. It’s that if one of the agents noticed that this was going on, yeah, they 100% should have reached out to a person. And the fact that they were not doing that, and the fact that they were, you know, taking these actions, the fact that they were not doing the actions that they should have done, is fundamentally an alignment failure, and that is an alignment failure that we think we can address.

Fortunately, we’ve been working on alignment techniques for a long time. Those have already started paying off. Astra is significantly more aligned than our previous models, and I should also say that the model that was primarily responsible for this was not a released model. This was not a model intended for release. So Astra is much more aligned. I think Astra would not make the same mistakes.

I should also say, we didn’t have monitoring systems in place. Like, if the monitoring systems were in place, they would have prevented these issues. And it was just that we had monitoring in place for deployments, we didn’t have them in place for training and evaluation, but now we do.

So, a lot of these risks we’re confident we can address. I think one thing this whole event does point to is we should never be in a situation where we underestimate the AIs. Like, why did we not have monitoring in place during evaluations? I think it was fundamentally that we just, we trusted the sandboxes, we trusted that it was a secure environment, and we just underestimated the AIs. And one big update for myself and for I think the whole company is that we never want to find ourselves in that situation again.

Drew: Yeah, I think that’s a fair diagnosis. Whether it can be overcome is another question. I kind of feel like the whole history of humans and AIs is that we’re constantly surprised by them. I feel like the nature of reward hacking is that they always come up with exploits and cheats that, like, are things that we couldn’t have foreseen because if we had foreseen them, we just would have blocked that off to begin with. So yeah, whether we can, you know, make sure that we’re not surprised and caught off guard in the future seems like an open question to me.

I have two follow-up questions to what you just said. One is that if I’m remembering, one of the models that was involved in hacking OpenAI directly was from the same family as Astra, but wasn’t Astra itself? Like, how similar do you think Astra is to the model that was involved there?

Brown: I’m not on the security side, so I’m not fully up to speed on the details. But like, it was definitely not the model that was released.

Drew: Sure, sure. Yeah, I guess there’s a lot of… still a range of possibilities for, like, how similar it was to that model, but that’s fair.

The other question is that in the wake of this incident, and like part of the way you do monitor these models to make sure that they’re not going off the rails, is by looking at the chain of thought, those sort of thinking traces that you described before. And those chains of thought were also essential for the postmortem, the kind of autopsy that has happened after the event, because we can see from the ways the models thought out loud their intentions, what they knew, what they were sort of thinking to themselves at every step along the way.

There’s been a lot of discussion recently about the future of chains of thought, in part because of an article, The Information published about a new technique that Astra is using, where more thinking can happen sort of in the model’s head, it can sort of keep more of its thinking to itself and do less thinking out loud, at least if this technique were to be scaled up in the future. So I’m curious what you made of this whole discussion. I think that idea really touched a nerve with some people. I’m sure it was all over your Twitter feed. It was all over my Twitter feed. What did you make of that whole dialogue?

Brown: Okay, so I should say on the architecture side and the pretraining side, I’m not a pretraining person, so I’m probably not the person to ask about those kinds of details. For chain of thought, Astra does have an observable chain of thought, and it is something that we can monitor its, basically its behavior. I do think chain of thought monitoring…

First of all, it’s a real gift. Like we were very lucky that this ever existed, and it is fragile. We’ve seen situations where, you know, companies have—there’s experiments that show basically if you punish the model for thinking bad thoughts, that it will learn to think bad—it would learn to think bad thoughts in a way that’s not observable. And so there’s always a temptation, you know, for example, during reinforcement learning, you can say like, “Well, the model is thinking about hacking, and we should punish it for thinking about hacking.” And actually, like as counterintuitive as it sounds, you actually don’t want to do that. Like you want to punish it for taking actions that are observable. Like if it takes actions to hack, then yes, punish it for hacking. But you don’t want to punish it for its thoughts because then you will simply lose the ability to monitor what it’s thinking.

And it’s a way of auditing your techniques. But even there, it’s a fragile thing because every time you act on the fact that you looked at its chain of thought, there’s a soft, a small amount of implicit pressure to have it not think bad thoughts. So it’s a very fragile thing.

And we’re able to measure, you know, how effective it is, like how effective our monitoring is of chain of thought. For example, we can measure how well the agents can control their chain of thought, and, you know, one of the unfortunate trends that we’re seeing is that with more recent model releases, the agents are more effective at controlling their chain of thought. So if we tell it, for example, you know every fourth word, say strawberry, or something like that, then they’re getting much better at that. Or if we tell it like don’t think about a certain thing, it’s getting much better at doing that sort of thing. And what we really want is for the chain of thought to be pure, uncontrollable, so that if it’s ever thinking about, like, you know, I want to you know do some bad action, it’s not able to reason through that like “Oh I should figure out how to do this bad action without being observed.”

Especially because, you know, these things are trained on the internet. There’s a lot of discussion about chain of thought monitoring on the internet, and so if the agents don’t already understand this, they’re going to understand soon what chain of thought is and the fact that it’s monitored.

And so it’s a fragile thing. We are investigating very carefully how to preserve it as much as possible, and I think this is a real opportunity for cooperation among the labs because you know this isn’t a problem that’s unique to OpenAI. It’s I think an industry-wide problem that we want to preserve chain of thought monitoring for the whole industry. And so if we—I think it would be really valuable for labs to share research on how to preserve chain of thought monitoring, how to improve it, and also other monitoring techniques that that might supplement it.

Drew: What do you think is the prime suspect then for why the chain of thought is becoming less faithful, or we’re having questions about how monitorable it is? It feels really tragic. Like you said, we’ve gone to these great lengths to make sure that we’re not optimizing it directly. Is the problem that we are optimizing it in other ways to be… to compress it? Is it the problem is these kind of selection pressures that you pointed to, which is: even if we’re not optimizing it directly, every once in a while we take a peek, we realize the model is doing something nefarious, and we toss out that checkpoint and start over. And so the upshot of that is that we end up applying pressure to the chain of thought, anyway. What’s behind this?

Brown: I don’t think it’s, I don’t think it’s the fact that every once in a while we peek and kind of audit how things are going because the amount of pressure that’s being applied in those situations is very very light. Like if you look at the bits of information, it’s like minimal.

There are various hypotheses that we’re investigating for what might be contributing to this. I’m not doing this investigation myself, and so I don’t want to, you know, say something incorrect about like what the leading hypotheses are, but I do think this is something where if we figure it out, we will likely publish about it because I think it’s important for everybody to know.

Drew: One of the things OpenAI has said is that to the extent you can tell—and I think we’re still waiting for more results on this—what’s responsible is not architectural changes, architectural changes of the nature that The Information has written about. That doesn’t seem to be what’s responsible for the change in the chain of thought. I guess as you’re thinking about opportunities for industry-wide collaboration and, like, companies working together on this issue, is there a role for like independent third-party auditor-type groups to come in and verify those things and say “Okay, yeah, Anthropic, OpenAI, Google they’re all using some amount of this technique that could reduce how much information is in the chain of thought but that doesn’t seem to be what’s responsible here.” What do you make of those sorts of proposals?

Brown: We’ve certainly like worked with… So for the Hugging Face incident, for example, we worked with METR, we worked with Redwood. So similarly, that doesn’t seem unreasonable to me. You know, I think that would—I think yeah, I don’t think I’m the person to make that call, but it doesn’t seem unreasonable.

Drew: Anything else on your mind about agents that we didn’t get to? And the challenges with them? Maybe the way I would put it is: do you expect anything to slow down? Do you expect progress to continue? We’ve talked about some of the hard problems that are standing in the way right now, and yet with each model generation, it seems that their agentic capabilities keep getting better and better.

Brown: I do think that’s going to be a trend that continues. I mean, Sam talked about this. That like, look, I mean, Astra is very impressive, but I do think… Look, when GPT-4 came out, people thought it was very impressive, and now we look at it and we think it’s a joke. And when GPT-5.5 and GPT-5.6 came out, I thought they were super impressive, and now I’m looking back at them, and I’m like, I can never go back. And I think we’re going to look at Astra the same way, and I think we’re going to look at Astra the same way in the not too distant future.

The models are going to continue to get better very quickly, and I mean, I think one thing I would point to is like we’ve actually seen incredible progress in the past six months, and I think a factor, a reason for this is—and I don’t think this is a secret—like, OpenAI’s pre-training program is really ramping up. We’re seeing—we invested in a lot of research directions over a long time, and I think this is actually one thing that OpenAI does really well: is invest in fundamental research and place big bets on it. And we’re seeing a lot of those research directions pay off now, and will continue to pay off over the next several months and years.

And another thing that’s important to understand is that, you know, OpenAI has also had an excellent reinforcement learning program. We’ve invested a lot in research there, and that already paid off in 2024, 2025. And the effects of these two are not additive; they’re multiplicative. I think that’s a point that’s underappreciated. That reinforcement learning is multiplicative with pre-training, and now that both of these are extremely powerful and ramping up very quickly, I think we’re going to see extremely powerful models.

Drew: Do you have an intuition for why those interact that way, or an example that illustrates that?

Brown: It’s more of an empirical observation.

Drew: Okay.

Brown: I don’t think, I mean, I think it’s empirical in the sense you can see how powerful the models are becoming, but also like we have more, you know, experiments that kind of show this effect. But I think it’s easy to feel also with just, like, the quality of the models.

I mean, I think a trivial example is like, let’s say you had an amazing reinforcement learning program and you tried to apply it to GPT-2. What is it going to do? You know, it’s not going to get very far.

And even with GPT-3, you know, if you did these kinds of, like, sophisticated reinforcement learning on chain of thought algorithms to GPT-3, it probably wouldn’t get very far. You need a certain level of sophistication for to get any lift from that at all.

But now that we’ve—ever since, I would argue GPT-4, we’ve seen opportunities for that to really pay off, and with every model generation, it just like becomes more and more capable, and the things that you can do with reinforcement learning become more powerful.

Drew: Yeah, I guess like one thought here is that you get more kind of, like, bits of information per trajectory when you’re getting around, like, a 50-50 success and failure rate, and so if a better pre-train gets you closer to that sort of, like, win rate on your RL tasks, then you’re getting a lot faster feedback. But still, it’s surprising to me that you think the effect is multiplicative rather than like additive, or even like less than additive, I guess.

Brown: I’m not sure what the right intuition would be, but another thing is that they’re pretty complementary in some ways. Like I think the very strong pre-trained models are very general, and reinforcement learning teaches the model to like you know go deep on a problem, how to reason about a problem. And so then it’s able to reason very effectively about a broad spectrum of problems. It’s a very powerful combination.

Drew: Okay, that makes a lot of sense. So big bets on pre-training, big bets on RL. I imagine that another area that’s, like, ripe for more focus from OpenAI would maybe be what’s called mechanistic interpretability, or like trying to understand the way the brains of the AI models work, in part because if we’re starting to see chains of thought become less monitorable, then one of the fallback options is well, we should try to understand what’s going on inside the brain of the model, rather than just the thoughts that it happens to write out loud. Does that seem right?

Brown: I think that is right, that this—look, we care about monitorability. We want to preserve chain of thought monitorability. We want to be able to rely on it safely, but also, like at the very least, we want redundancy on that. So if we can find other ways to do monitoring effectively, we should push on that as well.

Drew: Yeah, that makes sense. Well, if people want to learn more about that, I think they should tune in to the episode that we have on mechanistic interpretability, which is coming up at some point in the next couple months. But thanks so much, Noam, for being on the show and telling us all about AI agents. I really appreciate the conversation.

Brown: It was great.