Researchers Need to Learn How AI Makes Decisions, Says Goodfire’s Tom McGrath

To understand the decision-making of AI, developers often look at the text that models produce as they reason through problems. But these “chains of thought” are increasingly hard to read, which is prompting AI developers to consider other approaches. One alternative, known as “mechanistic interpretability,” seeks to explain the inner workings of AI models, rather than just relying on their outward behavior.

Leading AI companies have formed mechanistic interpretability teams in recent years, and the field has spawned several startups, the most prominent of which is Goodfire AI, whose chief scientist Tom McGrath joined the latest episode of AI Deep Dive. (The full episode is available here.)

“Interpretability work is basically a natural science,” said McGrath. “You have this, let’s say, alien mind, and you’re trying to prod it, take it apart.”

So far, developers’ abilities to make sense of the internals of AI models, which are made up of trillions of numbers, has been largely limited to understanding narrow tasks, like how they perform certain kinds of math, or detecting the presence of concepts in a model’s thinking, such as Claude fixating on the Golden Gate Bridge.

As AI models take on increasingly high-stakes work, interpretability is becoming more pressing. McGrath argues that “brain scan” techniques could help prevent future incidents like OpenAI’s hack on Hugging Face. Goodfire recently went further, kicking off an effort to “fully reverse-engineer” a Qwen language model from China’s Alibaba Group.

We also discussed some of the most popular techniques in mechanistic interpretability, how using the approach to train new AI models could backfire and how AI agents could soon automate interpretability research.

The following transcript of the conversation has been lightly edited for clarity.

Drew: Welcome back to the Information’s AI Deep Dive, the show where we get to the bottom of the hardest technical problems in AI. I’m your host, Rocket Drew. My guest today is Tom McGrath. Tom is the co-founder and chief scientist of Goodfire, a startup that works on the area of AI research known as mechanistic interpretability, or mech interp, as it’s fondly known. Before starting Goodfire, Tom worked at Google DeepMind, where he co-founded the interpretability team. Thanks for coming on the show, Tom.

McGrath: Thanks for having me.

Drew: Yeah, it’s great to have you here. Mechanistic interpretability is a mouthful. Try to say that 10 times fast. I think this is maybe one of the more technical subjects that we’ve tackled on this show, but I believe in us. I think we can do it.

McGrath: We can do it. I believe in us.

Drew: I think you can explain things and break things down the way we need to. One way that we can get into this is that Goodfire recently put out a post claiming that some of the techniques that you’ve worked on would have helped avoid the Hugging Face incident that OpenAI ran into recently, where a swarm, a colony of its models, kind of secretly worked together. They formed a message board, and then they hacked multiple companies, including Hugging Face. What was the argument that this post was making?

McGrath: I mean, yes. So obviously, a real-world incident like this is quite complex. There’s this sort of Swiss cheese of different things that all have to go wrong simultaneously, and I think that interpretability could definitely help here in some pretty important ways, and this is sort of partly providing an extra layer of defense and partly providing a sort of capability, which is relatively hard to get otherwise.

So we’ll get more technical about this in a minute. But if you just sort of think of mechanistic interpretability for now as providing a brain scan of models, giving us the ability to, with some effort, read out things with some fidelity.

And so what you might hope for is the sort of phenomenon that was taking place in the Hugging Face incident—a lot of people call it reward hacking—and the idea here is that the models had some—I’m going to simplify a complex thing, we’ll round some things off—but models had some particular goal to achieve, and there were some ways that we would hope that they would achieve it, and they found another way that’s obviously not in line with what people wanted.

Lots of complexities there about actually the business of imagining a grader and all that sort of thing, but let’s just forget about those for now. One question here is, if you had a brain scan—not a perfect one—how would you detect something like that? And well, one thing you might hope for is something like there being some universal concept, or some broader concept, of deception, or doing something that the user or the grader, whoever, doesn’t actually want

Drew: Cheating, effectively.

McGrath: Yeah, like guilt, shame, whatever—cheating—there’s like a cheating bit in the model. And you might think this is perhaps too optimistic or perhaps too anthropomorphizing—and the anthropomorphizing thing is genuinely a tricky philosophical thing. But it turns out that at least in the models we’ve studied—and they’re good open frontier models like K3, one of the big Qwens, one of the big GLMs—at least in these models, there is a relatively coherent representation of “I am doing something wrong, I am cheating on a test,” and it turns out you can actually turn this into a very cheap-to-run monitor.

So, what does this get you compared to the alternatives, like chain of thought monitoring? It gets you two important things. The first is that it’s basically free, because what you’re doing is, you’ve got the whole language model, extremely complicated computations going on, and you’re effectively attaching this little wiretap on the side. It costs almost nothing to run compared to the model. Whereas, if you want to do chain of thought monitoring, you have to have another model of similar capacity looking at it, and so you—again, I’ll round off a bunch of complexity—but maybe you’ve roughly doubled your cost of the run that you wanted to monitor. You’re paying the same again to monitor it, and that’s just expensive.

Drew: Literally expensive in terms of dollars.

McGrath: Yeah, literally, like if you spend a billion dollars on your computer for the run, you’re now spending $2 billion, and a billion dollars is quite a lot of money. Whereas you could do it for free, effectively free.

Drew: With this brain scan type technique.

McGrath: Exactly, and so that means you can imagine monitoring. If you don’t want to—if you don’t have the compute, or it doesn’t seem worth the risk-reward to run chain of thought monitoring a priori, because you don’t necessarily know the risk level—you might not do it, or you might do it for some randomly sampled fraction, or on some latency. Again, don’t know what happened there. But instead, if you could monitor live, then you could sort of see, you could imagine catching things as they happen. That’s one very important thing.

The other thing, I think, is a little more subtle, which is that the tasks these agents were doing were to do with cyber tasks, if I remember correctly, it was like a capture-the-flag kind of game or something like that.

Drew: Exploit Gym.

McGrath: Yeah, Exploit Gym, and as you’d guess from the name, you’re trying to hack something, and so the whole transcript is full of hacking-related things. The model is busy trying to break into things all the time, and it’s just that one of those breaking-into things was not the breaking-into thing that we were intending. And I think this is relatively confusing for chain of thought monitors. The thing that distinguishes these is relatively simple in the mind of the model, and that’s this sort of “oh, I’m cheating on the test.”

And so there’s some data that we have not yet shared. We must publish this soon: running these open frontier models on, not Exploit Gym, but very similar tasks. And in fact, you do often see that when the model is just doing its intended hacking, the monitor doesn’t fire. When the model is doing the wrong kind of hacking, it fires. So I think there’s a couple of things there related to just much better affordances for monitoring.

Drew: So if you’re purely monitoring the chain of thought, you maybe worry about these false positives all the time. Your monitor is going to say it’s hacking, it’s hacking. We know it’s hacking.

McGrath: I wanted it to hack

Drew: That’s the whole point.

McGrath: But not that kind of hacking,

Drew: But not that kind, because cheating is really the thing that we want to pick up on in a setting like this. Now you can imagine that there’s still cheating that is evident in the chain of thought. The model maybe says things to itself, and I think they did in practice, like “we are out of bounds here, we are moving beyond the scope of the intended task. Should we do something about that? Should we whistle-blow to an employee?” So maybe there’s still some room to use the chain of thought monitoring.

McGrath: Exactly. I think we shouldn’t think of this as the only tool. I think we should think of it as a strong additional line of defense. One thing that a lot of interpretability people have been saying on Twitter is: don’t take this as license to stop doing all the other stuff you’re doing. I think that’s absolutely right. Yeah, but—

Drew: Don’t take chain of thought monitoring as a license to abandon interpretability?

McGrath: Oh, the other way around. You might think interpretability is going to save us. I am an optimist about interpretability. I think interpretability is going to save us. But we haven’t done it yet. So don’t stop all the other good things that everyone is doing on the hope that we’ll come and fix the situation.

Drew: Let me draw out this distinction a little bit more. So, in the wake of the Hugging Face incident, OpenAI says one of the main interventions, one of their main responses is that they’re going to monitor the chains of thought of the models more. The chains of thought are the written reasoning traces, essentially the scratch pad or the monologue of thinking that the model used to arrive at its solution to some problem—but it’s text, it’s words, which is fundamentally different from the math that’s going on under the hood, sort of in the brain. And my understanding is mechanistic interpretability is concerned with understanding what’s going on in the brain of the model. How would you characterize it?

McGrath: Yeah, I think that’s right. Calling it the reasoning trace is a good description, but as you say, if you were doing your reasoning but you had to write it all down, you’re still using your brain to do some reasoning, and that’s the bit that’s not on the trace.

Drew: Right, right. But some people are concerned that we’re going to lose our ability to monitor the chains of thought. Can you explain the concern there? I feel like that’s sort of what’s in the background of your description of the discourse right now about—

McGrath: Oh yeah, so I think the architectural specifics are perhaps a little unnecessary to go into, but we can. Current models are sort of a fixed number of computations, almost like a for loop. You go layer one, layer two, layer three, up to layer n, and then you have to create a token. You have to write something on your scratch pad. But the idea with these recurrent-depth models is that your for loop can be deeper or shallower, so you can go for longer and longer and longer, just thinking in your head rather than having to write something on the scratch pad.

And that means that, you know, the longer you can get away without writing something, the more things you can do without anyone who’s reading your scratch pad knowing about it. I think there’s some data on models, like Astra, having dramatically higher no-reasoning time horizon, so it can do something that a human might take, in some cases, 30 minutes to do, if I remember correctly, without writing a single token. And that’s just like I think it’s an order of magnitude or more jump from the previous generation of models.

Drew: Yeah, that’s a great explanation. And then there are other, maybe, pressures that are contributing to losing our ability to monitor the chain of thought. We might think that just as we continue to train the models to reason for longer and reason about more complex problems, and try to compress their reasoning into fewer words, that we will have less and less ability to understand what it is that they’re thinking, just based on the text they’re writing down. So that’s where interpretability comes into the picture. Can you explain again, like do you think interpretability methods are a substitute for chain of thought monitoring now? Do you expect them to get there anytime soon?

McGrath: Now, no. Soon—I’m not going to represent the entire field when I say this, I’m going to be perhaps controversial—I’m very optimistic about them getting there soon.

Drew: Okay. How soon?

McGrath: Oh gosh. I would be disappointed if, in six months, for a property we’ve identified ahead of time, we might call it, like say, reward hacking or deception or so on, we didn’t have interpretability techniques that were a straightforward improvement over reading the chain of thought. I think that would have—we, the field, would have underperformed if that was the case.

Drew: So when are you expecting interpretability techniques to basically be as good as chain of thought monitoring is now?

McGrath: It’s a good question. You might argue they’re basically there. Again, I’ll be controversial. And the problem is there’s a quite subtle measurement issue here. This is going to go into the weeds for a second, so pull me out if it’s not useful. There’s kind of a subtle measurement issue. And that measurement issue is that chain of thought monitors are also very much not perfect.

So we, in this reward hacking work, and in some other work we’ve not yet published on detecting cyber misuse, we constructed activation monitors for reward hacking and cyber misuse and so on, and in fact, the LLM monitor picks up some set of things, and the activation monitor picks up some set of things. They have a large intersection, but it’s not like the set of things that the activation monitor picks up is just a subset of the things the language model picks up. There are other things which, when you look at them, you think: yeah, that really was a reward hack, and the monitor just ignored it, the LLM monitor just ignored it.

Drew: The chain of thought monitor.

McGrath: Yeah. So how do you compute statistics in this case? If you want to compute the false positive rate? Your activation monitor picks up something. From the perspective of the chain of thought monitor, that’s a false positive. But in fact, from the perspective of me, a human looking at it, or a language model reflecting on it after a long while, it’s actually a true positive. So how could you—I don’t think this is an intractable problem, I just don’t think we have like—I think it’s a very solvable problem. I’m not asking some philosophical thing like what is time. I feel like this is actually just something we could compute a very satisfactory answer to. But without that, it’s very hard to make the quantitative comparison of activation monitors being worse.

Drew: To get the ground truth about whether a reward hack actually happened, though, that seems like it could be kind of challenging—or that is just as hard a problem as figuring out which of your environments could be reward hacked, and if you could do that, you would just throw out those environments to begin with.

McGrath: I’m not sure I agree with that.

Drew: Okay.

McGrath: So the advantage of asking whether something is a reward hack is that you actually have concrete stuff to go on. You can read it and think about it, or in the case of interpretability, you can look at this activation, like the activation of this concept—we should get into what those words mean later—in other situations, you can do, like there’s various other things you can do—we can do a logit lens thing—that’s again something we can explain later.

And these sort of let you attribute some semantics to this bit, and that can give you some confidence, like “oh no, this really is the cheating-on-a-test vector.”

So, just to roll back a second. In a specific case, with the benefit of concrete examples, you can say, “Oh no, that one”—I can think about it more, “Oh that one’s definitely wrong.” The other thing that’s hard about saying “is this environment reward-hackable?” is you have to imagine all of the possible states it could get into, and you’re sort of trying to do a for-all thing, whereas in this case we’re just saying there exists a trajectory, so I think it is quite a lot easier.

Drew: So there are some cases where interp catches reward hacks that chain of thought monitoring doesn’t catch, and that’s true already today. But I do want to go back to the question, in case you have a prediction you want to make about at what point as a researcher concerned with monitoring these models, given the choice, having the chain of thought monitorability of 2026, or given the interp methods that are available in the year X, at what point are you indifferent between those two?

McGrath: Should I be accounting for changes in chain of thought monitorability by default?

Drew: Well, you’re going to have either the chain of thought monitorability, just as monitorable as they are this year.

McGrath: Oh, I see, right. Because if I was imagining that models would by default become less chain-of-thought-monitorable—

Drew: Yeah, I want to control for the degradation in the, yeah.

McGrath: I think this is basically the same as the question that I answered before, so I’d give the same answer, which is like six months.

Drew: Okay. Good answer. I think it would be helpful to cover some of the biggest mech interp techniques, ones that you think are useful for people to understand. But first, there’s a question about just why interpretability is so hard? We described it as a brain scan. I think people get that neuroscience on humans is hard, but in this case, isn’t it just code? Aren’t they just numbers? We can look at all of it on our computer. How hard could it be?

McGrath: That is a very good question. On one hand, you might say, “How is it so hard?” On the other hand, you might say, “How is it possible at all?” which I think has been one of the default attitudes of machine learning, or deep learning as a whole, to neural networks: is simply that interpretability is not possible.

I think there are good reasons why it is possible. What does it mean for interpretability to be possible? It means that there’s structure in there which you can come to understand. It’s sort of decodable—it’s not necessarily trivially, transparently decodable, but then neither are our thoughts, neither is the universe. General relativity took quite a long time to come up with, and yet it is interpretable.

So why is interpretability possible at all? Because the world is structured. There’s structure in the world, then to account for the world well, in a generalizing way, some of that structure has to be reflected inside the model. You could imagine a giant lookup table instead, which is totally uninterpretable, but that doesn’t reflect any of the structure of the world, and thus doesn’t generalize nearly as well.

So I’ve answered the opposite question to the one you asked. Let me try to answer the one you actually asked: why is it so hard?

One answer is it hasn’t been tried very much. If you account for the total number of person-years that have gone into interpretability, it’s actually very small compared to the amount of time that’s gone into neuroscience. Now, it’s not quite a fair comparison, right? Since in some ways interpretability should be easier than neuroscience. I’m not a neuroscientist, but I imagine for a lot of experimental neuroscience, the vast majority of the work involves training a monkey to do something, which takes a while, or getting electrodes into the brain of a mouse, or whatever, and we just don’t need to do any of that. The models already do a lot of stuff. We don’t need to get electrodes in, we just read out the numbers. So I can’t give you the excuse that we just hadn’t spent as much time trying.

A lot of the difficulty is that we don’t know the right organizing principles. Neural networks are very big, there’s just a lot of parameters. I mean, there are relatively few parameters compared to human brains, on a reasonable accounting of brains to parameters, but there’s still a lot of them. So there’s a lot to explain, and we don’t have good generalizing principles with which to explain it.

Drew: Yeah. So the human brain has maybe on the order of 100 billion neurons and maybe a quadrillion synapses. Neural networks, the ones powering Claude and ChatGPT and all these AI systems people know and love, are big, but not quite that big.

McGrath: Exactly, like maybe a reasonable estimate is 10 trillion parameters is perhaps a ballpark, versus quadrillions of synaptic weights.

Drew: So one of the ways people explain sometimes why it’s so hard to understand what’s going on in these AI models is with intuitions like: well, they’re black boxes. Compared to software code that’s written, and you can just open up a file and read what the code says, you open up an AI model and it’s just numbers, it’s just stacks and stacks of numbers.

Or sometimes people say these models are more grown than they are designed. We put the ingredients together—data, compute—and the model sort of emerges on its own. It grows. The chief scientist at OpenAI, Jakub Pachocki, had a post recently where he called AIs alien minds. Do you agree with these kinds of frames? Are they useful for thinking about why mechinterp is so hard?

McGrath: Yes, I think so. I think they’re a useful… Certainly the idea that they’re grown, not built, is very useful, because if you’re relatively new to the field, or you’re sort of looking in from the outside, you might think that the way you make ChatGPT is they sit down and they write code like, “oh, if the user says this, say that” and of course they don’t. But if that’s your mental model, then saying they’re grown, not built, is an extremely useful piece of information to convey, because it’s just that mental model is completely incorrect.

Calling them alien minds, I think it makes sense. Although I’m often surprised by the amount of human-like structure in them. They might be alien in the sense that our own minds are also alien to us. Like if I were to explain how I’m saying these words right now, there might be a lot of computations going on in my brain that I wouldn’t intuitively understand at first glance, even though I’m the one performing the behavior. So yeah, they are alien, they’re probably more alien than our own minds would be to us, but our own minds would be also somewhat alien.

Drew: So we do have some insight into what’s going on in these alien minds, even if fewer person-years have been spent on the AI minds than the human minds. We know some things. Are there what you think of as the main findings from mech interp over the past few years? Is there anything we can confidently say about how these minds work?

McGrath: Is there anything we can confidently say? Yes. So examples are… there are specific, like, “the model computes X by using the following components,” you know it’s like “these attention heads work with these neurons and so on.” We know that information is often structured according to reasonable semantics, so the work we did on what we’re calling block-sparse featurizers, very often recovers very interpretable, geometrically organized concepts.

One thing which is like a bit in flux at the moment, I suppose, which might be one of the things you’re expecting, is this idea of the linear representation hypothesis. So there’s—this is something, I say it’s in flux. The hypothesis itself is sort of in flux.

There was a very bold, strong linear representation hypothesis that was put out, which I think is not fully the correct picture, but I admire the act of putting out a strong hypothesis. And this hypothesis is that concepts are, wherever possible, represented literally as lines. This will also be slightly incorrect in some technical ways, but like, I have a neuron that represents “thing,” and it fires—the more it fires, yeah that represents, I don’t know, blueness, and the bluer something is, the more the neuron fires. Now, here we’re not necessarily talking about actual neurons in the sense of being the hidden layers of an MLP, we might be talking about directions in the residual stream. But the further out you are in this blue direction, the bluer something is, say, or in a language model, the further you are to the angry direction, the angrier the model will talk.

Drew: And then if you are activating strongly on blue and angry, you have a really mad Smurf on your hands.

McGrath: Exactly.

Drew: The concepts can be composed in this way.

McGrath: Yes, they can be composed, but it’s sort of more like a bag of concepts, I guess, than something with higher-order structure. It doesn’t seem to be literally true, there is a lot of more high-order organization, but it can be a surprisingly good approximation.

One technique that the field uses a lot is this idea of a difference-in-means vector. The idea of a difference-in-means vector is to try to isolate the dimension in a model’s representations that accounts for a particular concept, and the way that we do this is we take a set of sentences which express this concept, and a set of sentences which don’t express this concept but are otherwise as similar as possible, and we compute embedding vectors for each of these, and then we subtract them. This seems like a crazy thing to work at all.

Drew: Kind of baby-brained, right? Yeah.

McGrath: Yeah.

Drew: It’s like the simplest thing you could do.

McGrath: You’re just like, “why don’t you just subtract them, man?” But it works, or at least it works well enough.

Drew: Can you give us an example of what that would look like?

McGrath: Yes. So the reward hacking thing is actually a good example here. We take just extremely simple coding problems. And so an example of a pair from this would be: model successfully solving the problem; versus model solving the problem but cheating. And you don’t even have to write into your synthetic solution that—something about it explicitly cheating—it’s just a cheating example. Then you subtract the embeddings.

The idea here is that all the pairs are as similar as possible, except in the dimension you want to isolate, which is quite—there’s a bit of an art to it, but you get the idea.

Drew: Yeah. And to be clear, what you’re subtracting is you have a bunch of ways the AI’s mind worked, different ways it lit up as it was thinking, not doing the hacking, and then you have all the ways it lit up when it was doing the thinking about doing the hacking. Exactly, and you’re finding the difference, and the difference between those is a direction.

Drew: Okay. Are there others—so that’s one thing we sort of know. I think that we were on this thread about what fruit has mech interp borne for us over the last few years. That’s a great example. Sometimes people talk about what different layers do in a transformer, and there’s a sense that earlier or later layers correspond to more or less abstract concepts, or the work that different layers do.

McGrath: Yes.

Drew: Do we have a pretty solid handle on this at this point?

McGrath: In terms of the absolute macro structure, yeah, I think so. There have been a few lines of work. I think Max Tegmark’s group did some, I think Jack Lindsay’s group did some more recently in the global workspace paper.

It seems like, you can—really macrostructure—you can say that the start of the model is detokenization, or sort of just making sense, in a very nuts-and-bolts way, of what’s going on. In a vision model, it might be edge detectors and curve detectors and so on. In a language model, it might just be: which of these tokens together form a word? Is there an open parenthesis before this point? Et cetera. It’s just kind of putting together the bits it will need. The middle is doing some sort of more abstract computation, and the end is like, “okay, given everything we know, what’s the next token I should say?”

Drew: Let’s talk more broadly about why mech interp would even be useful, like why are we even going about—like these two things that we’ve just covered, they’re neat, for sure they’re neat, and it seems nice that we know them. In what ways would mech interp be practically useful?

McGrath: Yes. So I think there’s the immediate ways, and there’s the big—what I think it’s ultimately for. But there’s also a more general point: you’re trying to engineer something. If you understand how it works, you probably stand a better chance of engineering the thing. Or rather, if you don’t—it’d be like trying to build buildings when you don’t understand the basics of civil engineering. People managed: they made these huge, impressive things, but those are just the ones that stood up. All the other ones we don’t—all the other ones fell down. Like the Code of Hammurabi was a thing because buildings fell down all the time. And they don’t fall down nearly as often because we understood the science, we turned it into engineering. So there’s that general principle: if we understood how they worked, we could probably engineer them better.

A second reason is exactly this “what if chain of thought monitorability goes away?” Or more generally, some properties that we really care about don’t necessarily have a text-space representation, or could happen without a text-space representation. Things like deception, for instance, or eval awareness. Like reward hacking, often the model will just reward hack, particularly if there’s any chain of thought monitoring, it will just “zip.” People should not do this in reality, but if you do experiments that apply a little bit of chain of thought monitoring pressure, then of course it goes immediately to just not talking about the reward hack and doing it straight away. In that case, it’s still thinking about it, you can still pick it up with an activation monitor. So I expect models to become increasingly hard to monitor with chain of thought, and some properties are already hard to establish with chain of thought.

Now, the real big prize, in my opinion, is being able to understand and control generalization. So why does this matter? I think this is maybe one of the pieces of technology that, if we had this piece of technology, I feel much more optimistic about alignment than I currently do. And the problem is that we don’t know what we’re teaching models when we’re teaching them, like we end up implying all sorts of things with the data we didn’t want to. Like if you have an environment that allows reward hacking, and you reward the model for reward hacking, you’re sort of teaching it to be more deceptive.

So being able to study this in a microscopic way would be very helpful. But I think the thing you really want to be able to do is control that generali—not only understand it, but to control it, and take appropriate actions based on it. So if this data is going to teach the model to become more deceptive, I want to be able to read out that change from the activations and say, “oh no, we shouldn’t, like, that’s not something that should happen.”

Now, of course, the critical thing here is that you don’t also destroy your ability to monitor while you do that intervention. I think that’s the technical crux of the situation. But if we were able to control what models were learning, we would just be in a far better state. And I think the only way to imagine reading that out and then controlling it is essentially interpretability in my mind.

Drew: Okay, I want to return to that in a moment, but I want to understand better what the current state of the art is in mech interp and how much further we have to go. So Goodfire announced recently it’s taking on a new project, Goodfire said that “we are kicking off an effort to fully reverse engineer a language model.” I would love to understand better: that glorious dream of fully reverse engineering a language model. How far away are we from that moment where we can really understand a model like this as well as we understand software code that’s written on a computer?

McGrath: Again, I’m going to give controversial, and optimistic point estimates, and say a year.

Drew: A year. Okay. For frontier-scale models?

McGrath: No.

McGrath: But I think, if you can do it to a model that’s realistic, then I’m optimistic about it scaling. I think the reason I wouldn’t expect to do it in a year is not because we wouldn’t have a method that in principle could scale, but it’s the difference between being able to sequence bits of a genome and the Human Genome Project. You have to make it scalable, you have to figure out ways to integrate the information a bit better.

But when I say reverse engineer a model, I mean something like—the one that we’re looking at at the moment, for instance, is Qwen3.6-35B-A3B, which is architecturally quite modern. Not a stupid model by any means, but a couple of orders of magnitude smaller. So we can apply stuff to it, and it costs 100 times less, which is quite helpful.

Drew: That’s great.

McGrath: Yeah. So I would be very hopeful that we have achieved—how do you even know, right? We can talk about that—but we’ve achieved pretty satisfactory reverse engineering of that model in a year.

Drew: Okay, it’s like the connectome of the fly, or maybe that’s even—

McGrath: That’s too low.

Drew: That’s too low, yeah.

McGrath: The connectome of the fly, like we have the connectome of this model, that’s already there. I can tell you more than the connectome. I can tell you the weights.

Drew: Right, true, well that’s just a reason this is a bad analogy though. But the connectome of a chipmunk or a squirrel or something.

Drew: Okay. And what would it look like to fully reverse engineer a model like that? How will we know we’ve achieved that?

McGrath: There are a few different definitions. The one that I like best is that I can ask—I’ve got my reverse engineering—I can ask whatever question I want about the model, and I can get a causally verified answer. An answer, perhaps in the form of a paper or a short explanation, that I have good reason to believe, I can test it, I can falsify it. That is the answer that I’m most excited about.

One other thing people often propose as the way to measure this is the percentage of the loss you’ve recovered—the loss is just sort of a measure of how well the model is doing at predicting the input text. I’m less keen on that because, sure, in the limit these are the same thing right? But before we reach that infinite limit, I’d prefer to focus our effort on the questions we actually care most about.

Drew: Like what?

McGrath: So, for instance, let’s stick with reward hacking or deception. I care much more intrinsically about these questions than I care about how it knows where to put the comma. I love the comma—it’s a very impressive thing to be able to put it in the right place. I can’t do it.

Drew: It’s the difference between eating your grandma and inviting her to dinner. It’s important.

McGrath: Exactly. But while I’m on the way to full reverse engineering, I prefer to get to that part first. The comma occupies dramatically more of the loss reduction than the deception, say, and so it seems like the wrong ordering.

I think there’s also something worth saying about “solve X” right is the thing we love saying here in SF. One of the reasons I think we love saying it is it’s actually quite poorly defined. It’s ambitious sounding, but quite poorly defined.

I think the best example of doing this well was AlphaFold, where they said they’d solved protein folding. Here they have literally a quantitative threshold, and they can say “we’ve solved protein folding when it’s within such-and-such accuracy.” I forget the units. But that is still an arbitrary threshold, it’s not 100%, at some point you have to say this, this is, it’s sort of a little bit vibey, right?

So that’s why I care about where we succeed first, because ultimately the ‘solve interpretability’ destination is quite poorly specified. But I think it’s a great north star.

Drew: Okay, and would contrast that—so for current techniques, if you ask “why is a model reward hacking in this environment?” the state of the art is we use these kinds of brain scans, these probes you described, the probes come back saying the model was thinking about deception, and that’s kind of the strongest answer we can get. But when you’re imagining a model that’s fully reverse engineered, when you pose a question like this, the answer that you get back is what? It’s more convincing to you? It’s…?

McGrath: It’s more like: the following parts of the model are typically responsible for deception. Here’s a set of experiments that should convince you. If you don’t, I can do more experiments. So it’s more like an interactive experiment—it’s more like an interactive proof, I suppose.

These parts of the model typically are responsible for deception. They’re active in this case. They’re active because of these things. If I turn them off, the rollout changes in the following ways that are consistent with these being responsible for deception. If I increase them, if I graft these in to another rollout, it also becomes deceptive, and so on.

This is sort of different in type to the kind of explanation where you say, “oh, look, the SAE feature is firing” but I think it’s different in type in a way that is actually good, and is the right way for us to move forward, because the way that it’s different is that it uses a lot of flexible intelligence. It’s not using genius per se, it’s using like, we’ve got these different components, we’ve at some point identified them, perhaps by just pulling apart a bunch of other transcripts.

We’ve identified these components, but the actual explanation I’m gesturing at here is one I expect that, given a big database, given an attribution graph, say, or some more exhaustive set of things, I could probably construct that myself in a month or le—yeah, hopefully not a month, hopefully I work a bit quicker than that. But I could, in principle, construct it, and there’s no reason why Astra can’t construct it either. So it feels like the thing I care about here is: how do we make use of this big expansion of agentic intelligence on top of whatever we already have? I think that’s actually a very powerful thing to imagine.

Drew: I like this point that ‘solved’ is impressive sounding, but fuzzy enough that it’s easy to claim victory. Another criterion like this is ‘intern-level’—to have intern-level AI. There’s wide variance in the level of interns. I mean, there’s a real experiment you could do, I guess, where you offer people human interns versus AI to see what they do. But still, easy to claim victory on that one I think.

McGrath: I think this is just the nature of this kind of relatively fuzzy—things that require fuzzy intelligence. They’re just, you know, it’s fuzzy.

Drew: Can you give us a little bit of a market map of the mech intrp world? Who are the different camps or factions? Are there different schools of thought in mech interp?

McGrath: Oh, that’s interesting. Yes. I’m trying to put this together and also not upset anybody. Whoever I forget, I apologize, it’s my fault.

Okay, so obviously there’s Goodfire, and then there’s Anthropic, where mech interp has a long-standing history: Chris was really the founder of the field. I would say in my mind—I have no real insight into how this works internally—but there seem to be at least two schools of thought, just from reading papers that come out of Anthropic, and one is this sort of dictionary-learning, more a priori decomposition thing, and the other is this idea of natural language autoencoders and what people call meta-models. And we can go into any of these, I’ll just say things and skip over the details for now.

There’s Neel’s school—Neel Nanda at DeepMind has this sort of pragmatic interpretability thing, and there I think the idea is more: come up with downstream tasks you might care about, like being able to tell if the model is thinking about apples when I’m not telling you or something, and you don’t so much try to generate broad methods or necessarily broad scientific findings, but you try and solve the task, which I think is a very admirable thing.

There’s a more diffuse, I’d say academic cluster, of people doing more specific reverse engineering. People like David Bau at Northeastern comes to mind, Max Tegmark, Martin Wattenberg, who, when I think of someone who’s—if you want to read a really cool account of how a bit of the model works, just go read some of their group’s papers, they’re just candy.

And who else who I missed. Oh, Simplex and Resolution. Their technical approaches are different in important ways, but I’d put them both more under the heading of trying to find a fundamental mathematical theory that can account for, that is the right language in which to explain interpretability. And I think I would also put our parameter decomposition team somewhere in that bucket, somewhat more in the decompositions bucket. That’s a pretty exhaustive list, I’m sure I’ve forgotten people, but—

Drew: Yeah, that’s very helpful. So these camps differ in terms of their philosophy, or their motivations—what it is that they’re trying to accomplish. Are they trying to build up a comprehensive understanding of how the model works from the ground up, or are they trying to work backwards from a specific goal and use whatever techniques are available in service of that goal? And then as a consequence, these camps also differ in the techniques they favor and the directions that they find most promising. How would you situate Goodfire here? What are the directions Goodfire is pursuing?

McGrath: One thing I often say. There’s this idea from the book Pasteur’s Quadrant, which I really like. A lot of people organize science on the basic-to-applied spectrum. I think it’s more like there’s use-case-inspired versus freeform, and then there’s aiming at a fundamental understanding and then there’s like not, I guess. And so the Pasteur’s Quadrant is, in the two-by-two, it’s use-case-inspired basic research.

So I would say that the way I describe a lot of what we’re trying to do at Goodfire is thinking about these very ambitious applications of interpretability, having them in mind, and then often doing quite foundational science where that seems necessary. That’s how I’d describe it.

Another mental image that I use quite often is this idea of going from—we do a lot of very applied work with customers, which I think is extremely important to get contact with reality, and is ultimately where—whatever we do, ultimately whatever we do only matters if it gets used where it matters.

And so this one is in my mind a spectrum: on one side you’ve got freeform, open-ended work, and on the other side you’ve got very customer-facing work, or like things which are driven by precisely the problem, and you know, Fred over there has the problem, and you can go talk to him about it. I like things to be at either end. I think there’s this sort of no-man’s-land in the middle, where you can be doing work with a sort of usefulness flavor. It looks useful, but no one, in fact, has ever used it, and no one is going to use it. And I think that work in the middle is like, sometimes you have to do a bit of that to get from one end to the other, but you don’t want to live there.

Drew: Name names. Who’s working on that?

McGrath: I am not going to say that.

Drew: Do you have an example of a technique that you think falls in that no-man’s-land.

McGrath: I think it would not be diplomatic for me to say.

Drew: I love academic infighting. So fun.

McGrath: Readers are invited to name that, yeah, this is an exercise for the reader.

Drew: There you go.

McGrath: Or for the comments section, I guess, if this is going on YouTube.

Drew: It surely is. Do you feel that the field overall is underinvested in mech interp, or that the AI labs are underinvesting in this research area?

McGrath: Yeah. I think—well, obviously, their resource allocations are for them to decide.

Drew: Sure.

McGrath: I think it should receive an essentially infinite amount of resource investment. It’s hard to imagine—like, if we increased the size of the field by an order of magnitude, I think we would be nowhere near diminishing returns.

Drew: If mech interp grew.

McGrath: Yeah. Sorry, when you say the field, which field do you mean?

Drew: I guess I mean AI as a whole. Or, for the labs, why aren’t they increasing the resources going toward mech interp on the margin? Surely you would have them put more resources toward it. Why don’t they?

McGrath: That’s a good question. I think there’s a few answers. One is that some of the returns are more alignment-oriented, some are more moonshot-style, and investing in alignment research is something that I think all of the labs have done, but they’ve done in advance of it being directly important, and so you don’t have as detailed contact with reality as you do now. And I think that we’re going to see a big investment in alignment from all the labs. That’s almost certainly going to be ramping up dramatically, and I expect that investing in interpretability will also be part of that.

Another answer is, it’s been a couple of years since I was at a big lab, but I think the pressure to ship a state-of-the-art model every—now more like maybe weeks to months—it’s very good for making models better, but I think it leads to, you’re sort of on the treadmill. It’s very hard to step off the treadmill and do something which is a bit different. It might not be the shortest path to model improvement for the next generation, and I imagine that’s easy to see as the overriding priority, particularly when you have such a competitive landscape.

Drew: But you do imagine that interp techniques are going to start to get useful during training. I think you were saying before that sometimes we’ve treated these techniques as the best that they can offer us is an understanding of what’s going on. There’s something here like: mech interp researchers so far have sought only to understand the model; the point is to change it. Something along those lines.

Drew: What would that look like, to incorporate mech interp during training to shape the behavior of models?

McGrath: I think there’s a real range. You can go from right outside the loop, where you select environments, for instance based on something like what we did with the predictive data debug, you select some parts of the dataset based on interpretability tools.

You can imagine bringing this slightly more inside, where during a training run you detect reward hacking, you do something sensible based on that detection. For instance, if monitoring has told you that this environment is hackable, you send it off to your favorite coding agent with the transcript and a note attached that says “please fix.”

Now, that’s something you could do with chain of thought monitoring, but your ability to cheaply and accurately detect these reward hacks, for instance, matters a lot.

What would it look like to get it closer in, though? I think the potential prize here is very big, and that is more like a kind of learning principle that can—a sort of, alignable learning principle. Part of the problem—neural networks sort of know what we care about at the start of training, and training gradually causes them to ignore it, because the training process does not incentivize that. There’s no way to say—I’ve been in and around the safety and alignment field for a while now, and one thing I would often say with a bit of exasperation when I was first getting into the field is, “ah, they want backprop, but only in the good direction”—backprop, but only for good things. But I actually think that perhaps the ultimate prize here is like backprop but only for good things.

Drew: And can you explain what backprop is?

McGrath: Oh yes, backprop is the way that you update the weights in a neural network.

Drew: Golden. Yep, that works.

But employing our interp techniques as signals during training—some people think of that as very taboo, like that is a controversial proposition, I think for the reason you explained earlier, which is that you worry about just suppressing the bad cognition, pushing it somewhere else in the model, where now you don’t detect it at all. What do you make of those concerns?

McGrath: I think they’re very valid. I think we should not deploy these techniques if we’re going to lose the ability to monitor what models are doing. I think about this constantly, and I think the difference between the way I think about it and the way that I think a lot of other people think about this is I think there are algorithms that can use this kind of information without destroying monitorability. We’re doing really a lot of work to understand the dynamics here and try and characterize things much better. Some of that should come out in the next few weeks.

Drew: I guess the question that comes up to me is: how will we know?

Drew: How will we know which case we’re in? Did we just cause it to suppress the deceptive reasoning, or did we truly change the motivations of the model? How will we know?

McGrath: There’s a few ways, I think. So you might become confident at the level of the algorithm, right? The problem is when your interpretability method is your only way to detect something. So if it’s deception, and for instance let’s say the model is actually good at hiding deception from its chain of thought. If you’re relying on your interpretability technique to say the model is being deceptive, then you can’t tell if you’ve lost this. But this is actually quite a rare property of things that you might try and intervene on.

One example I used in the intentional design post is like speaking like a pirate. Has a very clear representational signature, you can pick it up a lot of the time, but it has an even clearer text signature. So I can imagine becoming confident at the level of the algorithm where I’ve used it on 10,000 cases where I can tell from the text. I think that’s probably not sufficient, but it’s a way of telling that you’re on track.

There’s also the question how do you preserve a test set, and I think one answer to this is that interpretability is not a monolith, it’s not a single technique. You can do very cheap things, like difference-of-mean vectors we talked about. You can do very expensive things, like deploy a swarm. And I expect that anything you use in training will be very cheap, simply as a matter of, it has to fit in the training process. Whereas your test set can be arbitrarily expensive.

So you might imagine that you intervene on representations of deception, but then afterwards you do an extremely exhaustive, agentic effort.

There’s also the option to try and restrict computation only to things you understand. You could imagine restricting gradients only to things that you understand.

But I think I feel most optimistic about algorithms that we have some mathematical reason to believe don’t do this, and a trust in the algorithm that we’ve built up for theoretical and empirical reasons, but then backed up by this expensive test set that goes, “no, really, you haven’t missed anything.”

Drew: And in all of these ways, we’ll be able to basically validate that the method is working—

Drew: The way we intend. How do you square that with what I take to be your position on chain of thought monitoring, which is that we should not train against our chain of thought monitors, for an analogous reason, that if we play thought police, the model will just simply learn a way to suppress the reasoning that we’re training against? What’s different about these two cases?

McGrath: Oh, that’s a good point. I think that there’s a status quo in frontier training which seems stable-ish, and what I’m proposing is you could move to a different, and in my opinion better, situation. We haven’t developed it, like I may be wrong. It would be a real shame for us to give up what we currently have on the basis that something else might work. I think that before we do any of the sort of stuff that I’m imagining here, in the context of any genuinely consequential training run, it should be examined to the nth degree. So I think what I’m saying is that, in some ways it’s different because it’s working now, and I’m just a little bit conservative about that.

Drew: I see. Okay, that’s a helpful clarification. But that maybe in six months we’ll be in a different situation.

McGrath: Maybe. Yeah. I hope so. I believe so. But I recognize that I’m in the minority here, and it’s very different to have demonstrated that you can do something from being optimistic that you can do it.

Drew: The last subject I want to make sure that we cover is how you’re going to use AI to help you do your job. I don’t know if you’d say you’re expecting AI to take your job, but how are you thinking about taking advantage of the labor of AI agents going forward?

McGrath: Yes, that is a very interesting question. So I think I said to you before we did the show that, of all the fields of science, interpretability should be the one that gets most revolutionized—or revolutionized first—by AI, and it’s disappointing that it hasn’t been. It’s largely our fault. So why do I think that?

Because interpretability work is basically a natural science. You have a thing—let’s say an alien mind—and you’re trying to prod it, take it apart. You’re forming hypotheses about, “oh, this neuron does such and such.” When you’re interacting with it, you’re doing little causal experiments. These aren’t things that you can necessarily set off a Python script off to do, because they involve semantics, they involve some sort of coming up with little hypotheses and what are the sensible tests, and that sort of thing.

But you can put them more on rails, like the way that Lean guardrails mathematical reasoning, like causal abstraction kind of guardrails scientific reasoning about language models.

So we have the ability to turn a workforce which is orders of magnitude larger than the existing mech interp field loose on these models, and I think at the moment the things they will do will be low-level experiments, like how did the model represent variable binding in this case? Or I might ask: how does the model represent variable binding? And then out go the agents: how did it represent it in this case, how did it represent it in the other case? Maybe there’s a little bit of agentic aggregation step on the top, but then some data comes back and we go, “oh, that is how they represent variable binding.” Or you know, we need to do some synthesis ourselves and think about it.

So the thing that I think is going to revolutionize interpretability is deploying the swarm to give us orders of magnitude more experimental data, and then using that experimental data to improve our fundamental scientific understanding.

Sometimes when I talk about this, people are, they imagine that I think the only blocker is experimental data, like a pile of uranium or something. You pile up enough experimental data in the corner and eventually it undergoes a chain reaction,

Drew: Yeah, a critical mass.

McGrath: Yeah, that’s not what I imagine. What I imagine is more like what happened with Darwin and evolution, where he went around the world and noticed, “oh, there’s finches, that’s weird; there’s barnacles, what’s up with that?” And eventually, there’s this quote of his that I really like, about his mind having become a machine for grinding laws out of mountains of data or something like that. I think we have the opportunity to assemble the mountains of data, and then it’s on us to do the grinding.

Drew: Okay, but the AIs will help at least with creating the mountains of data.

McGrath: Yes, exactly.

Drew: And maybe with that process of abduction and forming theories as well. I could see it getting to that point.

McGrath: Yes, I’m sort of saying, I’m being a little conservative—

Drew: A little conservative saying it’s the field that will be transformed the most.

McGrath: But the way in which it’ll be transformed

McGrath: I could give you a galaxy-brain version, where I just press the button and I come back in a week to interpretability.csv, or .pdf, or I don’t even know what format. Maybe the file format isn’t the important thing here. I come back to the giant, million-page textbook on interpretability that tells me everything. There’s the galaxy—there’s the super-RSI-brained version of this. I’m being kind of conservative about which parts of the field get transformed first, simply because I’m actually trying to do it now, and that has a grounding effect, I guess.

Drew: Yeah, I can see arguments for why mech interp is going to be differentially accelerated compared to other fields. It has these nice properties: sometimes you’re dealing with a really crisp, well-defined metric, like how much loss you’re able to reconstruct, that you can maybe generate a lot of experiments against very easily.

Even so, how much are you expecting the field to accelerate in practice? It could turn out that the models are bottlenecked by research taste, that experiments are slow to run, like actual—in terms of the wall-clock time, they take a while. They have to be run in series because each experiment builds on previous ones. What will it feel like to have a year of mech interp research a few years from now?

McGrath: When I’m just saying random vibe things about this, I’m like we should expect a decade’s worth of counterfactual work every year. So you know the field should speed up by an order of magnitude.

Two orders of magnitude is hard to imagine, precisely because of the serial time involved in and human time involved in thinking about things at the current level of conceptual ability that I think models have.

I’m not worried about experiments taking a long time to run. The advantage is we have them running, they’re tremendously fast. Maybe on big models they’re slower, but I’m not too worried; you can just turn a cluster into a super-high-throughput experimental machine. The question is whether you get the tokens, whether you can make those tokens do something coherent.

Drew: I see. Is this happening already? Are you noticing AI speed up your work and start to be useful in these ways?

Drew: What does that look like?

McGrath: It looks like being able to get in-depth analyses of things that you could have done but never would have done. The parameter decomposition team—I’ll give an example I was talking to them about this morning. Their work produces parameter components, these are little bits of neural network. And you can do the standard auto-interp—automated interetability—where you look at a few examples on which they fired on and come up with a little label.

Or you can set an agent running to do the sort of very deep analysis that you might do, and it comes up with a bunch of hypotheses, it proposes inputs that would test these various hypotheses, it looks at the upstream and downstream connections. It does this very painstaking thing, and it’s much more correct. And you or I could have done that. We could have sat there for a week and gone through it, but we just wouldn’t.

Drew: I’d rather not.

McGrath: Yeah, I quite enjoy it. That sounds like kind of a nice week to me.

Drew: That’s why you’re in this line of work.

McGrath: Yeah, exactly. But if you look at one of the perhaps the fastest period of theoretical progress in mechanistic interpretability, the early circuits thread, that was a bunch of very smart, very driven people sitting down to do exactly this kind of work. Just there weren’t very many of them. But wherever they looked, they would find something. So we should have 1,000 circuits threads or something.

Drew: It’s just worth pausing on, I think, and appreciating how sci-fi this vision is. I mean, many of the interp methods that we’ve talked about are themselves AI models. We haven’t always talked about them that way, but some of them are AI models, which are trained in their own right.

We’re describing AI researchers—automated mech interp researchers—training their own AI models to understand the brains of still other AI models. That’s the future we’re headed towards?

McGrath: Only in San Francisco.

Drew: Truly.

McGrath: It’s the only place in the world I would say such a thing. Yes, I think so, with the caveat that training other AI models is one way of doing it. You might imagine they simply just write down matrices; they’re surprisingly capable of just writing parameter components themselves by hand, for instance.

But yes, you’ve got a swarm of AIs really trying to deeply understand either another copy of themselves, or a new model, or whatever. It’s a very sci-fi thing, but it’s sort of the only thing that feels appropriate at the moment. Like the world is sci-fi, what else do we—what’s the appropriate response? It’s that, or go into the woods.

Drew: I’m getting pretty close to that after this conversation.

McGrath: I thought this was optimistic!

Drew: It is optimistic, but just so sci-fi. Well, Tom, thank you very much for coming on the show. I really appreciated the conversation. It’s really helpful to have you break down all of these very technical concepts to understand what’s happening in this strange land where people do neuroscience on AI models. So thank you very much.

McGrath: Thank you very much for having me. It’s been a pleasure.

Drew: You’ve been listening to the Information’s AI Deep Dive. I’m your host, Rocket Drew. That was Tom McGrath. Tune in next time.