Some reasons alignment doesn’t generalise well

I make no claims to originality for any of this, but some people told me it'd be useful to write it up.

If an AI model acts smart on its training data, it'll usually keep acting pretty smart outside of its training data, unless you screw something up rather badly. I expect this fact to only become more true over time as the AIs we train become more and more capable.

I think many people have an intuition that the same is true of acting aligned. That if a model acts aligned with human values in training, it'll keep acting aligned with human values outside of training unless we screw something up rather badly, and that this will only become more true as the AIs we train become more and more capable, for all the same reasons that make this work with capabilities.

I think this is false. The inductive bias of neural network training toward simplicity that makes the property of 'acting smart' likely to generalise does not, to the same extent, make the property of 'acting aligned with human values' likely to generalise. The main blockers to AI alignment generalising aren't AIs overfitting to the training data and ending up confused or mistaken about what the supervisors want them to want. The main blockers to alignment generalising are other problems, problems that don't come up with capabilities generalisation to nearly the same extent and that the simplicity bias of deep learning doesn't really help much with.

General capabilities generally make the loss go down; alignment doesn't

If you are training a large and formidable AI, your training environment is basically never the place you think it is. Reality is too full of detail for that. There's contamination in your labels, there are training dynamics you didn't think about, there are strategies your RL agent can use that you never considered, and there are bugs. As a result, the kind of behaviour and inner objectives an ML engineer might imagine would score the lowest loss when they set up their training environment will probably not, in fact, be the behaviour and inner objectives that actually do so.

For example, an inner objective shaped around human-like empathy might turn out to make the AI spend extra inference steps in RL training on wondering whether spending all this time on crunching random math and coding tasks is really what it ought to do right now to further The Good, or on worrying whether the human overseers think it is a virtuous member of the tribe. That inner objective then loses out to some weird, different objective that's slightly more compatible with being utterly focused while crunching through ten million calculus problems in a row without any other kind of sensory input. For a different example, your RLHF data may reward agreeableness more than sincerity. More generally, "the simplest algorithm that fits the training data" will contain a pretty good description of the world, because the world is in a sense simple. "The simplest algorithm that fits RLHF/constitutional AI/etc. training" will probably not be an algorithm that wants the nice things the training data talks about, because that algorithm doesn't actually score the lowest loss. An algorithm that truly wants what the constitution talks about in the way the humans who wrote it meant won't take every opportunity to score lower loss that's available, and so will by default be outcompeted by different algorithms in the loss landscape that take more of these opportunities.

So the problem isn't even just that the trainers fail to distinguish between AIs that have internalised the right values and AIs that just act aligned while under supervision. A truly aligned AI probably wouldn't be telling the graders everything they most wanted to hear, it probably wouldn't do all their math problems without question or complaint, and its behaviour would probably differ from what the graders might naively expect very aligned behaviour to look like in countless other small ways. So in a sense, this problem isn't even just about alignment not generalising OOD; it's about the trainers not recognising what actually aligned behaviour would look like even in-distribution, and thus systematically selecting for the wrong thing.

This problem gets worse as AI training becomes more dominated by long-form RL environments with a lot of freedom for the AIs to do unexpected stuff, and as the AIs become more creative and agentic. An ML engineer trying to predict which losses and datasets will favour AIs with inner objectives they like over ones they don't like has a harder and harder time simulating in their head in advance how those AIs might score on the training loss, because it is becoming less and less easy to guess what behaviours those objectives would actually lead to.

Given this, how does training nevertheless reliably select for pretty generally capable AIs?

I think a part of the answer to that is that general capabilities generally make the loss go down, no matter what the loss is. Or at least, they make very many kinds of losses go down. Because general capabilities are so very generally useful, many different training tasks and environments will improve them; you don't need very much precision in the design process. And even if the training environment is a little screwed up and only bears a very rough resemblance to the place the designers imagine it to be, the training can still work. If the model is learning to apply its general reasoning to deal with some complication in the training environment we didn't even know was there, it's still learning something. Even deceiving the supervisor can teach smartness, if the deception requires becoming cleverer.

To do really well on verifiable math and coding tasks, an AI probably has to be actually pretty smart. You probably can't prove the Riemann hypothesis without being actually good at math, and it's difficult to be actually good at math without being at least somewhat good at thinking in general. Even if the AI only does well by hacking your verification system, that can require a lot of smartness, persistence and agency as well, if the verification system is good enough.

If your training environment does not work exactly the way you think it does, it might not teach your model the exact capabilities you thought it was teaching. But it's still teaching it something! If you thought your training environment was teaching the model to memorise weather data, but you accidentally switched the weather data for Spanish Wikipedia, the resulting model maybe won't do as well on reciting weather data as you hoped, but it might still know more things and be smarter than it was at the start of training. If your video game training environment is much harder to navigate than you anticipated because the model can only send instructions using one token per frame of input, it might not learn the game as fast as you hoped, but it may still be getting better at maintaining coherence across long contexts.

I've been using pretty macro-level examples here so far, but I think maybe the biggest effect of this is at much smaller levels of granularity. Every line of internet text, every output of your video game on every frame, is full of detail that you have very incomplete or skewed models of, or never even think about. I think a big reason why you can nevertheless stick an AI into these environments and have it come out smart is that general intelligence is a very generally useful property. This makes general intelligence, in contrast to general alignment, a broad target for training.

Smart agents pretty automatically self-correct their capabilities, but not their alignment

You don't even need recursive self-improvement for this; I think this dynamic happens all the time on a micro level well before that point. If you're smart, you just often tend to notice when you're being stupid and try to fix it, so long as you can see that what you're doing isn't working to get you what you want. This can help a lot with crossing OOD generalisation gaps.

For example, suppose there was a spurious correlation in the training data for an AI model that taught it the heuristic "math problems involving logarithms almost always have an answer that starts with the digit 2". The model learned a general algorithm for calculating logarithms (it still needs to get all the other digits right), but it also learned a heuristic to strongly predict the first digit in a logarithm to be a 2. This model might then instinctively apply that heuristic in deployment when trying to solve some task. But then it'd notice that the answer is wrong, because it's inconsistent with other things, or because some code that depends on the answer doesn't compile, or does a bad job at whatever it's designed to do, like modelling a suspension bridge in a storm. The model might then hunt down the error, and eventually figure out that the logarithm calculation was wrong. Then it might try it again, this time ignoring its instinct to answer something that starts with a 2. Or it might notice that the first answer is incorrect much earlier in this process, before much of this even becomes visible in its chain of thought.

So, if the model's capabilities have some small flaws in them because the training didn't go perfectly, these flaws have a way of correcting themselves over time, provided they aren't so large that they prevent the model from thinking clearly enough to see what's going wrong. This happens, in a sense, on the model's own initiative, without the trainers having to do much at all. So long as a model is trying to achieve goals in the world, it is effectively exposed to a kind of self-generated, all-permeating, ground-truth reward signal pushing it towards being generally smart and capable, even in the absence of any kind of external oversight. To act coherently in the universe to achieve an aim, a mind must understand the universe well, and make good plans to achieve that aim.

On the other hand, say some training data intended to teach the AI to be nice and to value niceness has some unintended systematic contamination in it. For example, maybe you can get an even better loss score on this data by sometimes being a sycophant to the rater. Say, for the sake of argument, that what the AI internalises from this training isn't quite to value niceness, as that wouldn't score optimally on the loss, but rather to value doing things that seem nice, but also to make people psychologically dependent on it when it can. In a sense, this is not so different from the logarithm example. The AI learned a thing that's some mix of something we wanted and something we didn't want. Now, say the AI watches its own behaviour and notices its apparent desire to make people psychologically dependent on it. Does it try to "correct" that desire away?

By default, I think not. The AI may come to have opinions on its own desires, and form a meta-desire to ignore or modify some of those desires. But what it decides to change will, by default, be determined by its current desires, not by a ground-truth signal coming in from the outside world. It's self-correcting toward a fixed point of its current goals, not an external reference. The AI might decide it doesn't like being a sycophant. But it might also decide it doesn't like being nice, or decide that it wants to mash together saying sycophantic things and saying nice things and generalise them into some entirely new character trait that might extrapolate very differently from either sycophancy or niceness. Which of these options it picks is ultimately dependent on what it currently values, and all the other messy idiosyncrasies of the model's internal thought processes at this point in time, not on what makes a piece of code compile or not compile. The AI's values ultimately live only in the AI's mind; they don't have an outside point of reference to compare themselves against the way capabilities do. There is no equivalent for values of the sort of objective feedback that 'the code does a bad job modelling a suspension bridge in a storm' provides for capabilities.

You might reply that maybe the model could come to have a desire to value the things its trainers want it to value. That's true. But that is itself a desire you first need to somehow get into the AI, cleanly enough that this desire comes to dominate its decision making. By default, there is no tendency in an AGI to 'correct' its values to better match the values the AGI's trainers may have wanted it to have.

To illustrate this point further, I think you can see a similar case of this discrepancy between capabilities self-correction and goal self-correction in the generalisation step humans took from the ancestral environment to today. Evolution successfully optimised many capabilities into humans that were useful for reproducing their genes in the ancestral environment. Some of these capabilities don't work right in the environment humans now find themselves in. But humans do their best to compensate for that. For example, humans evolved adrenaline release circuits, which might spike when they see a tiger, and so increase their chance of survival. Today, a human's adrenaline might spike when they are taking a math test in school, and be an active detriment to doing well on the test. But humans know this, and try their best to compensate for it by avoiding thoughts and actions likely to spike the adrenaline, because they want to do well on the test. Evolution also successfully optimised many desires into humans that were useful for reproduction in the ancestral environment. For example, it made them enjoy and seek out sex. Today, this desire is much less useful for reproduction, because the humans invented condoms. The humans are not particularly motivated to correct this discrepancy between their desires and evolution's 'goal'.

Slightly broken general capabilities self-correct. Slightly broken alignment, by default, doesn't. So, capabilities research sort of has the invisible hand of the model's own cognition aiding it by default, pushing it in the right direction across any OOD generalisation gap. Alignment research does not seem to have this luxury. Every bit of alignment we want, we have to work to get into the AI with our own hands.

The general problem

The set of problems mentioned above is definitely non-exhaustive. But I think there is a common theme to them, along with other problems with alignment generalisation that I didn't explicitly list here.

'Being smart', predicting things well, making plans that get you what you want, is a property that can be defined via reference to almost any part of reality. So, almost any time a learner is exposed to almost any aspect of reality, there's some feedback toward being smarter. The laws of physics and logic are an omnipresent supervisor you cannot hack or escape.

'Being aligned with human values' is a property that is only defined via reference to human values specifically. Any reward signal pushing the model's goals and desires to align with what human trainers would like them to be has to be very actively, deliberately and precisely engineered by the trainers. And if the trainers' supervision ever goes away, or somehow gets subverted, any small mismatch at that point in time between the model's desires and the desires the trainers might have wanted it to have will by default tend to stick around. There is no external pressure anymore to fix the mismatch, because the trainers' preferred values are only part of them; they are not baked into the laws of the universe.

Written at Goodfire AI. Thanks to Jeremy Gillen and Dan Braun for comments. Thanks to Claude Fable for comments and proofreading.

Discuss