Labs’ current alignment plans seem terrible

Tl;dr: The frontier AI companies’ implicit alignment plans center on rushing towards RSI with something like current models. This plan is (at best) contentious, and many believe it will have catastrophic consequences. Nonetheless, it is the track we are on. It is therefore important to collect evidence which helps us anticipate how this will go.

—

Note: This post is based on conversations inside the Arcadia Impact alignment team, thinking about what constitutes “good” work in third-party alignment research.

What are the labs’ current alignment strategies?

In practice, the labs’ playbook for producing aligned models looks like the following:

1. Shape the model’s character. Both Anthropic and OpenAI treat alignment training as a question of which character or persona the model takes on. Ideally, planting a ‘safe’ character into the model will generalize to all of its other actions.
2. Apply extensive RL pressure. Somewhat independently of the above, the labs do extraordinary amounts of RL on the model. This increases its capabilities, but also encourages reward-seeking behaviors and metagaming, and may ultimately lead to scheming and other forms of instrumental convergence.
   1. They also might apply various techniques here such as inoculation prompting, confessions, and the like.
3. Race towards prosaic RSI. The intend to use this model for automated research to improve both capabilities and alignment for subsequent models. People in labs say that this is necessary to make sufficiently fast progress on alignment, conditioned on the race conditions.
4. Evaluate and fix alignment via trial-and-error. The labs try to catch alignment failures via dedicated evals, but many alignment issues are found through intended and unintended deployments.

This plan seems terrible

The above strategy appears deeply flawed.

To start, capabilities milestones continue to arrive faster than anticipated, and our alignment techniques are already unreliable. Recent events suggest that optimizing for task success seems to inherently come at the expense of alignment. Although the models are evaluated for how well the RL and alignment properties coexist, it’s unclear how to do these evaluations.

Prosaic RSI seems unlikely to close this gap. We have no evidence that RSI dynamics will differentially benefit alignment over capabilities and, indeed, we have reasons to believe that they won’t. On top of this, automated alignment research asks us to trust the outputs of models whose alignment we’re studying. Consequently, alignment may fall further behind as automation ramps up.

However, we may not notice future alignment failures. Under RSI, models will change quickly and dramatically. The intuition which has formed from today’s AIs may become antiquated, and we will likely understand even less about the alignment properties of the future systems being deployed. This is compounded by the fact that people will be removed from direct training, relying on other models to monitor and summarize the progress.

In either case, as we approach ASI, we will no longer have the luxury of finding out about models’ alignment properties via trial-and-error.

This is all made worse by the race dynamics, which allow labs to excuse poor safety practices in favor of rushing on capabilities. Regardless of whether prosaic RSI favors alignment or capabilities, it may compress timelines in ways we are ill-equipped to handle. In addition to this, the race encourages:

• An engineering mindset. Labs seem to be treating alignment as an engineering problem to be patched until the evals look good, even if we don’t have scientific methods for determining whether better performance on alignment evals is a result of actually-better alignment. Importantly, we don’t have reasons to believe that this alignment whack-a-mole will scale to ASI.
• Hastiness. Labs seem to consistently neglect basic safety procedures due to the speed pressures they are under (speed pressures which are, in large part, self-inflicted).
• Lack of transparency. Frontier labs guard their IP closely and work to preserve as much secrecy as they can. Even when they’ve involved third party auditors, this seems to be primarily reactive (i.e. in response to incidents), done on the labs' terms, and may come with various restrictions that water down the resulting information value. As such, there is often no publicly legible evidence of egregious misalignment or of labs’ alignment practices.

This default trajectory seems catastrophic.

And, unfortunately, the tools we have been developing to contain these catastrophes (monitoring and broader AI control frameworks) seem insufficient to handle the above issues.

Consensus-building is hard

People broadly believe the current plan is crazy. People inside the labs and those that have left say that this idea is crazy. Many are adamant that things have to slow down.

The fact that these statements coexist within organizations that are nonetheless pursuing RSI suggests—among other things—an absence of consensus around the risks. Some reasons this consensus has not been achieved could include:

• Key figures in AI and politicians remain unconvinced about catastrophic risk from RSI.
• Laypeople distrust corporations by default and see their calls for regulation as dishonest; this motivates them to find alternative explanations. 
• Frontier AI labs provide reasons to distrust their motives, e.g., lobbying against AI regulation, making little effort to communicate basic facts on AI risk, and being insufficiently candid about incidents.

We need better evidence

We think the most effective way to achieve consensus is to produce more legible evidence of misalignment under labs' current strategies. In particular, many of our current arguments about alignment and RSI are either speculative or insufficiently convincing. Labs have large incentives to believe that their alignment techniques work and our outputs must be able to stand up to that level of scrutiny.

It's important, though, that this evidence is not propaganda! Said otherwise, we should open-mindedly study the consequences of the labs' alignment strategies and report them openly.

Such evidence would be directly useful for multiple parties:

1. Embedded evaluators. Third party auditors are up against the clock when going into labs: there are many unknown unknowns and the auditors’ attention is their most valuable resource. Compiling evidence about misalignment sources under RSI should help auditors direct their time and efforts more efficiently.
2. People in labs. Many people inside labs are trying to do good safety work. Similar to the embedded evaluators, their time and resources are limited. To the extent possible, the alignment research community should make it easier for safety people in labs to raise issues.
3. Regulators. Assuming that we should prioritize regulation around RSI, it is valuable for regulators to have maximum context on what specifically they are regulating.
4. The general public. Many people in the general public say they "didn’t ask for this" with respect to AI. However, for reasons related to transparency and takeoff speeds, people are unlikely to even have a full picture of what specifically they did not ask for.

How should alignment research change?

More concretely, we claim that future alignment research should embody the following principles:

• Reason about alignment holistically. Alignment is an emergent property of the entire process by which models get developed and deployed. We should reason about it holistically. In particular, it's most important to test whether "alignment techniques" are effective in the broader context of the full training pipeline and deployment ecosystem, especially if they have elements which would likely induce misalignment. This is in contrast to simply testing alignment techniques in isolation.
• Contend with pessimistic assumptions. It is important to highlight the ways alignment strategies will break under pressure—there will be a lot of pressure!
  It's easy to study alignment strategies under optimistic / easy conditions and come away with positive results. However, the same alignment strategies could fall apart under more stringent / suboptimal conditions. As we develop AGI, it is not sufficient to only show cleaned up, best-case examples of techniques working (like this, this, or this).
• Be as transparent as possible. Consider two possible worlds:
  • Alignment is falling behind capabilities. In this case, society deserves to know as soon as possible. Improving the state of public evidence can build consensus towards a coordinated global slowdown / pause, and buy ourselves time to improve alignment for real.
  • There’s a way to make alignment keep pace with capabilities. In this case, society also deserves to know! If we discover a silver bullet for alignment, then the underlying science should be shared broadly, so that organizations developing powerful AI outside the frontier labs do not build heavily misaligned models.

Total research transparency is also a key component in Plan A from AI 2040.

Specific directions we're excited about

Given the above context, the Arcadia Impact alignment team is currently working on the following technical topics. We would appreciate feedback on whether people think other things should be a higher priority:

Studying how RL affects motivations. It seems like scaling up reinforcement learning leads to many of the failure modes we see in present-day models: reward hacking, motivated reasoning, metagaming, and perhaps instrumental convergence. We require a better understanding of whether these are resolvable issues, as well as a better understanding of how to differentiate principled solutions from patchwork ones.

Stress-testing scalable monitoring. The volume of work done by agents will grow by many orders of magnitude in the future. We'll increasingly rely on scalable monitoring protocols to tell us what's going on. It would be very bad if we were overconfident in how well these protocols work. This will become especially important once we start deploying models on tasks that lack an easily-verifiable ground truth.

Anticipating how prosaic RSI goes wrong. Eventually, models will be put into automated research harnesses even if they're misaligned, either by accident (misalignment was not detected) or intentionally (labs decide to compromise on safety). What are the ways this could play out? This is the least conceptually clear of our directions, due to high uncertainty about what research automation actually looks like. Still, we think it's valuable to do carefully designed experiments comparing aligned and misaligned models in order to surface some broad phenomena.

Thanks to Brandon Riggs, Joe Hardie, Sid Baines, Angel Martinez, Moritz Pfau, Francis Rhys Ward, Emma Mouliere, Jonathan Bostock and Alex Beat for comments/discussion.

1. We use the term prosaic RSI to refer to Recursive self-improvement via AI automating the existing AI development pipeline, rather than via direct self-modification.
2. For instance, it is unclear to us whether and how labs currently do alignment RL.
3. Members of the AIS community have already spent years internalizing the relevant evidence. They may find it personally easy to dismiss arguments . As a result they might think it not worth the effort to rebut in detail. We think this is a mistake; there is a vast inferential gap to be crossed, and repeating the old arguments ad nauseum is unlikely to be effective.
4. Some common objections to AI risk from RSI are:
   1. If our attempts at alignment have been unreliable thus far, models self-aligning may be our best–shot.
   2. All of the doubts about the future are speculation, so there is no legitimate reason to pause current RSI efforts.
   3. Labs have incentives to notice if their internal alignment is truly insufficient.
   4. Even though the current pace of AI development is untenable, we would need cooperation with X entity to enact meaningful safeguards and we are unlikely to get that cooperation.
   5. AI will not reach superintelligent levels or be intelligent enough to be beyond human oversight / affect the real world.
5. This feels related to big(Θ) and big(O) notation in theoretical computer science, where you describe an algorithm’s runtime under average- and worst-case conditions. In our case, we should have a culture of evaluating alignment under modal and worst-case lab conditions.