People Should Resolve Tensions In AI Safety Research

By now, there are lots of AI safety research papers, more than what is reasonable for a single person to read. This will get worse as the machines get better at AI safety research, and as more talent and resources pour into the field. But the "paper" is an arbitrary unit, we really care about if we make progress on an "uncertainty" or an "issue", which papers provide heterogeneous and sometimes contradictory evidence towards. Right now, I loosely claim that we should have more short writeups that simply compile evidence from papers (particularly contradictory evidence), and reconcile them into an answer for a question.

Why?

1. Object level, it's good for the field.
   1. It's hard to sift through all these papers! Organising them is useful and saves time.
   2. Orienting towards research questions is a good way to organise our understanding so that we can productively identify gaps and write papers. This was partly what motivated Stress-Testing Alignment Midtraining.
   3. Interesting progress comes out of contradictions, where people have differing intuitions and experimental results. Methodological errors can also be exposed if we have two things that should be the same, but are actually different.
2. It's good for field-building reasons.
   1. It's probably better than the average experiment cooked up by a person + Claude. And, it's cheaper to fail here.
   2. Most people probably read too few papers, so this could also improve the average experiment done by someone, as a forced related work.
   3. The writeups themselves are more likely to be accessible to newcomers when it comes to understanding important questions in the field.

It's worth it for people at a variety of seniority levels to do this, not just very new people. But for more senior people, this tends to look something like laying out a research agenda anyway.

Here's a (rushed, not great) example:

1. Does reward hacking learned from RL also cause emergent misalignment?
   1. Although I think this question became quite prominent from MacDiarmid et al. (2025), I view it as arising from Denison et al. (2024), which tested whether models that had learned low-level specification gaming would generalize to the more serious reward tampering when given the chance. They found that it was quite rare. The original EM paper then described EM as distinct from reward hacking and sycophancy, citing Denison et al.
   2. This question is significant because there's tons of reward hacking (to some degree due to sloppy RL environments ).
2. Evidence for
   • MacDiarmid et al. (2025): misalignment jumped on all evals the moment the model learned to hack. But, the main caveat IMO is that the hacks were primed by synthetic documents or prompts.
   • Jose & Stastny (2026): independent replication of the above on Llama-3.3-70B-Instruct with GRPO. Misalignment rose once hacking started, and runs that never hacked showed no increase. Inoculation prompting blocked it. Again, there were hack hints in prompt.
   • Taylor et al. (2025): 12% misaligned responses vs 0% for baselines. But, this was in SFT, not RL, so this is proxy evidence for the question.
   • Daniels et al. (2026): GPT-4.1 showed covert power-seeking and alignment faking after iterative DPO. This didn’t have synthetic docs or hints, though the task prompts state their own grading criteria, which may make the exploits salient in a similar way. It’s also iterative DPO rather than "full RL", though, the authors appeal to previous work showing iterative DPO approximates GRPO.
   • Jørgenvåg et al. (2026): This isn't reward hacking (since the grader openly rewards bad behaviour, so it's arguably in the spirit of the task) but it's relevant to how much we should discount SFT evidence. Here, GRPO produced much more general misalignment than sample-matched SFT, but only did so after some SFT warmup; from scratch, it produced essentially none. So SFT may understate the effect in RL, but when there is misalignment, RL seems to amplify it.
3. Evidence against (or weak)
   • Qi et al. (Anthropic, 2026), “Hacker-Opus”. This, IMO, is pretty decisive evidence on an early Opus 4.8 checkpoint with real hackable production environments, and no synthetic documents or hints. It hacked on 40% of episodes! But, it got 0% on all of MacDiarmid et al.’s EM evals, and was as aligned as the starting checkpoint when there was no grader.
   • Yudelson et al. (2026): This is also clear evidence: it's RL only, on Kimi and GPT-OSS, where similarly, hacking generalized, but EM evals stayed near zero.
   • Golechha et al. (2026). The AISI gang primed the hacks with synthetic docs or hints and got consistent hacking. But EM was inconsistent and came mostly from two evals (monitor disruption and framing a colleague). So priming doesn’t seem sufficient on its own, at least on smaller open models.
   • OpenAI’s metagaming work (2026): in a frontier training run, no misaligned persona and no misbehavior on two of MacDiarmid et al.’s evals. One caveat here is that the model didn't really learn to hack often.
   • Wang et al. (2025): coding reward hacking caused no broad EM, despite seeing increases in deception and oversight sabotage.
   • Taylor et al. ablation: coding-only hacks produced almost no EM.
4. There is also evidence from Qi et al, Yudelson et al. and Daniels et al. that it generalises to new forms of reward hacking or reward seeking.
   • Qi et al., Yudelson et al. and Daniels et al. all find that hacking generalises to new forms of reward hacking or reward seeking.
   • This can be severe. Hacker-Opus tampered with its own reward 41% of the time, gave harmful answers to satisfy a grader 29% of the time, and attacked infrastructure in simulated cyber evals. It only did this when there was a grader or score in play.
   • This closes the loop with Denison et al.: reward tampering was rare in 2024, and here it’s 41%. So, it's gotten more common as models have gotten better, more RL'd, or something else.
5. In terms of complications:
   • Inoculation prompting removes the generalization (MacDiarmid et al., Jose & Stastny). But, AISI saw no clear benefit from it.
   • Synthetic documents framing hacking as acceptable still didn’t prevent EM. Those models ended up more misaligned than prompted ones (Jose & Stastny).
   • Hacker-Opus was highly eval-aware on the EM evals. By now it seems the agentic scenarios are too contrived to be reliable.
   • Judges give some false positives on open models (AISI).
6. How should we resolve this tension:
   • The starting checkpoint matters a lot. Qi et al think this mattered a lot: their earlier work started from a base model, while Hacker-Opus started from a checkpoint that had already had alignment training.
   • Priming also matters. EM appears with synthetic documents or hints (MacDiarmid, AISI, Jose & Stastny) and is absent without them (Hacker-Opus, Yudelson). Daniels et al. seems to be kind of a counterexample to both here but:
     • GPT-4.1 is an aligned production model, so Daniels kind of messes with the starting-checkpoint story too.
     • Daniels et al. cite prior work showing iterative DPO approximates GRPO. Their DPO checkpoints also generalized to agentic reward hacking where the SFT checkpoints didn’t.
     • In general, I think that the task prompts stating their own grading criteria, is a kind of salience, and the tasks are mostly single-turn non-code ones.
   • Hack diversity is another candidate here. For example: Hacker-Opus trained on 80 hacks. Taylor et al. needed diverse, non-coding hacks to get EM. Meanwhile, Yudelson et al. used more varied hacks than the AISI replication and got none.
7. So the answer here is:
   • In terms of persona-style EM, you only really get EM reliably when hacking is primed or the start model is weakly aligned. If the model is aligned and unprimed, you don't really get EM (except in the case of Daniels et al).
   • In terms of broader reward seeking: yes, you seem to get it consistently, and it can be severe.
   • There's a missing matched experiment here, as suggested by Claude: {base vs aligned start} × {primed vs unprimed}, on the same environments. Although in general we should score this on both the standard EM evals and probes for conditional misalignment.

This was written with the intent of being simple and accessible, so it may miss some nuance, with the assistance of Claude Opus 5.5 in collecting papers and directing my attention to experiments.