Monitorability evaluations and policy proposals

There has been a lot of discussion recently of policy proposals around monitorability, such as Redwood's proposal and my counter-proposal. I'd like to say a bit more generally about how I'm thinking about monitorability. Note that nothing here is an official Anthropic position; these are just my personal thoughts.

First, a categorization. I currently think about four different types of monitorability:

1. Control Monitorability: the extent to which Control and monitoring techniques would likely catch catastrophic sabotage actions, like escaping a sandbox or setting up a rogue internal deployment. We can measure this via reference to sabotage evaluations like LinuxArena or any evaluation that measures the model’s capability to take dangerous actions like escaping a sandbox while looking to a monitor like it's doing some innocuous task (like a cyber eval).
2. Training Monitorability: the extent to which reward models, monitors, and oversight in training would likely catch misaligned or undesirable actions in training, such as sneaky/subtle reward hacks.
3. Auditing Monitorability: the extent to which alignment audits, evaluations, and test deployments would likely incriminate a systematically misaligned model (e.g. a schemer)—as well as the extent to which monitor evaluations would likely detect a model's propensity to collude as a monitor. We can measure this via reference to the model's capability to consistently distinguish between our most realistic alignment audits / monitor evals and real internal deployment cases where it would have an opportunity to do something catastrophic.
4. Diffuse Monitorability: the extent to which humans working with the models generally understand what the models are doing and why they are doing so. Unfortunately, I think this is the one we least understand and so I'm mostly going to ignore it for now until we figure out better methodologies for evaluating it; I'll maybe try to say more about this later when I feel like I understand it better.

Second, some notes on trustedness, which is currently my biggest conceptual uncertainty with how to think about monitorability. A basic fact that is true about at least all of the non-diffuse forms of monitorability is that you should be able to in some sense get a perfect score by just using the same model as the monitor and policy. That is because for any bad thing that the policy can think to do, if the monitor is the same model, it too should be able to come up with and detect that.

So if we ever want to be able to use monitorability as a meaningful red line, we need to have a very good answer as to why that might be insufficient. The standard answer in the literature is trustedness: you need to be able to trust that your monitor is aligned and thus not just lying to you in its monitoring. Conventionally, the idea is to separate models into "trusted" and "untrusted" based on whether you have previously made a confident alignment safety case for them. Then, you can either just evaluate monitorability relative to a trusted monitor, or try to come up with monitoring schemes that combine trusted and untrusted monitoring in some way that makes it hard for the untrusted monitors to collude (of which there are many such schemes, a simple one being to just run both the trusted and untrusted monitors together and then if the untrusted monitor ever doesn't flag something that the trusted monitor does flag, you get very suspicious that the untrusted monitor is sandbagging).

However, this basic setup has some issues, probably the biggest one being how to deal with white-box monitoring. The conservative thing to do is to just treat white-box monitors as untrusted, though in theory you might think they should be trusted because it's not clear that a misaligned model would actually be able to spoof a white-box monitor looking at its own activations. However, I think this idea is pretty untested and relies on white-box monitors picking up on deep features about the model's intent, which it's very unclear to me that they actually do—for example, if you train a white-box monitor on prior examples you were able to detect (rather than e.g. some sort of unsupervised discovery of an intent-laden direction), that leaves open the possibility of a generalization where the classifier learns to only detect the sort of things that you would previously have been able to detect but doesn't detect subtler things that a smarter policy might be able to do.

Given all of that, I would propose the following methodology for how to do (non-diffuse) monitorability evaluations:

1. Use Auditing Monitorability evaluations to determine trustedness. The idea here is that the main argument we make against systematic misalignment in Anthropic's Risk Report right now is based on models lacking the robustness to avoid giving themselves away in auditing if they were really trying to implement a systematically misaligned policy. Thus, so long as we feel good about Auditing Monitorability, we should feel good about trustedness from a systematic misalignment perspective. There is still a question, though, of whether rare cases could induce monitor collusion in a non-systematic way—to deal with this, you can also evaluate the extent to which the model can tell apart your monitor evals (that are looking for monitor collusion) from real monitoring; this is a bit tricky, but should definitely also be a part of your Auditing Monitorability evaluations to establish trustedness.
2. Evaluate Control and Training Monitorability relative to the best trusted monitor(s) that you deploy in practice (either in internal deployment or in training, respectively). I think once you have a reasonable notion of trustedness, "best actually deployed trusted monitor" is the correct conservative choice to evaluate with respect to.
3. Do at least prompt iteration elicitation if not RL elicitation. In theory, the nice thing about all of these sorts of monitorability evaluations is that they are capability evaluations and thus should be possible to do in a way that isn't susceptible to sandbagging by a systematically misaligned policy (see: "When can we trust model evaluations?"). To accomplish that, however, you have to actually make sure you're doing good elicitation. The most basic form of elicitation here is: a) hold out some subset of your monitorability evaluations, b) iterate a bunch (e.g. by throwing a large swarm of agents at the problem) on finding the prompt that gets the best (as in least monitorable) performance across the non-held-out subset of your monitorability evaluations, and c) check transfer to the held-out evaluations. Even better than that is for (b) to actually involve doing RL against your monitors—which at least in the Training Monitorability case is pretty realistic given that if we're talking about e.g. a reward model then the model should actually receive gradients to subvert it in training.

Finally, some thoughts on monitorability-based policy proposals. The big thing I'd like to see is a combination of a) red lines for absolute levels of unmonitorability that labs should not cross and b) measures that constrain the degree to which labs can do things that degrade monitorability even before reaching those red lines. (a) is in some sense the best thing to do here, though I'm somewhat nervous about it in isolation given that it doesn't constrain actions at all now, which is why I'd prefer it to be coupled to some sort of (b) that meaningfully binds all along the way.

I've already written about what I'd like to see for (b), which is to evaluate the quotient of unmonitorability/capabilities and reject any substantial model change that makes that quotient—what I'll call the "monitorability externality"—too high. The idea is that a monitorability externality would be a change that makes monitorability even worse than would otherwise have previously been predicted for a given level of capabilities. For actually evaluating this quotient, I suggest simply using the methodology I described above for evaluating all the different types of unmonitorability, and then something like ECI for capabilities.

In addition to proposals like that which result in constraints along the way, I also think that ideally we'd be able to enshrine some hard monitorability red lines as well. I'd propose both Control and Training Monitorability red lines—with Auditing Monitorability implicitly present to the extent that it constrains the trustedness of the monitors that you evaluate the others with respect to, as above. Specifically, I'd suggest something like:

1. Labs should not be able to internally deploy models that can reliably evade detection when attempting dangerous sabotage actions like breaking out of a sandbox, poisoning data, or sabotaging an alignment experiment.
2. Labs should not be able to train models that can reliably evade detection when attempting reward hacks in training.