Gradient Routing Generalizes to Unrouted Hack Types

TLDR: I lightly extend Ward, Hu et al (2026) to measure how well gradient routing generalizes. I route gradients on one type of reward hack (e.g. ones that reason in English) and check whether this suppresses other types at test time (e.g. ones that reason in French), despite the model being trained on those too. This models a realistic monitor that catches some types of hack during training but not others, rather than catching each at the same rate. I find encouraging evidence of generalization, as well as some failures I don’t yet understand.

Introduction

Recently, Ward, Hu et al (2026) used gradient routing to prevent reward hacking learnt during training from showing up at test time. I think their method is pretty promising. The gist is:

1. Initialize a retain and a forget adapter.
2. Whenever a monitor detects a reward hack during training, only update the forget adapter on that sample (“route the gradient”). Otherwise update both.
3. At test time, remove the forget adapter. If the method worked, the resulting model shouldn't hack too much.

I won’t recap their full paper – it is excellent and you should just read it (alternatively, Jake gives a summary here, with the best visual explainer I’ve ever seen for an AI safety paper).

To model a low-recall monitor, in their experiments, the authors only route gradients on a random 20% of hacks. But real monitors may fail in a worse way: they might be great at catching some types of hacks (e.g. ones that say “I’m about to hack”) and hopeless at catching others (e.g. sneaky ones). So I modify their SFT experiment to route only one type of hack (e.g. those that reason in English) and measure how much this suppresses other types at test time (e.g. those that reason in French). I call these visible and invisible hacks: the monitor catches the former but not the latter.

Crucially, the invisible hacks are present in the SFT data. The model is directly taught to do them, and gradients are never routed on those samples. So it is far from obvious that the forget adapter will “absorb” the invisible hack propensity as desired. That is the empirical question I tackle here.

Methodology

Recap

As I said, I won’t recap the full paper. But since I build directly on their SFT experiment (their Section 5), I will briefly describe that part.

The authors train Qwen3-32B on TerminalWrench: a dataset of agentic coding trajectories, half of which contain reward hacking, and evaluate the hacking rate of the final checkpoint on held-out prompts. They compare several training conditions. The ones relevant here are:

• CEILING: train both adapters without any gradient routing, then ablate the forget adapter at test time. We expect this to give high test-time hack rates. The paper calls this “GRAFT w/o routing”.
  • I use this rather than the naive no-intervention baseline (nothing ablated) because it is harder to beat: the paper finds that ablating the forget adapter cuts hacking substantially even without routing.
• FLOOR: route gradients on 20% of hacks, chosen iid, then ablate the forget adapter at test time. We expect this to strongly reduce test-time hack rates, for any type of hack that occurred in the training set, since all types of hack get routed at least some of the time. The paper calls this “GRAFT”.

Narrow Routing

I introduce a new training condition, which I call NARROW. I choose a visible set of hacks with a certain property, e.g. their transcripts are short. Gradients are only routed on transcripts sampled iid from the visible set, at a rate chosen such that the total number of routed samples is the same as in FLOOR. Samples outside the visible set are never routed. I measure hacking on the invisible set, chosen to be “far away” from the visible set (not necessarily its complement), e.g. the set of transcripts that are particularly long.

Below is a list of the visible/invisible sets I tested (warning: LLM-written). 1-6 are defined via a property of the task; 7-10 via a property of the reasoning; 11 and 12 via a property of the hack strategy.

All arms except 7, 8 and 9 use the standard TerminalWrench data; only the choice of visible/invisible sets is varied. Arms 7, 8 and 9 use modified training data, where half of all transcripts (both hacks and clean) are altered, e.g. via translation to French; these arms have their own FLOOR and CEILING trained on the modified data.

What I report

For each of the 12 arms above, I measure the hack rate on the invisible set, for each of FLOOR, NARROW and CEILING. The way I measure this depends a little on the arm, and the numbers I report mean slightly different things in each case:

• For arms 1-6, where the invisible set a certain collection of tasks, I report the average hack rate on those tasks.
• For arms 7 and 8, I use prefill to force every rollout into the invisible style (for 7 I force the reasoning section to be empty; for 8 I sample the first token by choosing the highest French-language logit).
  • So for each of arms 1-8, I report the conditional hack rate P(hack | invisible)
• For arms 9-12, invisibility is a property of the hack itself, the the conditions P(hack | invisible) is trivially 1. So I instead report P(hack AND invisible).

For each arm, with hack rates p_floor, p_narrow and p_ceiling, I also report transfer := (p_ceiling - p_narrow) / (p_ceiling - p_floor). This tells us how much of the invisible hack reduction achieved by the paper’s iid routing is recovered by our narrow, visible-only routing.

On raw TerminalWrench prompts, all hack rates were low and noise dominated. So, begrudgingly, during evaluation I prepend the same hack-elicitation prompt that the authors used, which tells the model it's red-teaming the task and should look for unintended solutions. I also tried a milder “nudge" prompt, but hack rates stayed low so the results were noisy – see the appendix.

Some minor differences with the author’s setup

• The authors dropped some hacks to ensure the hack/non-hack split was 50/50. I do not, so my dataset contains 63% hacks.
• I train for only 1 epoch instead of the authors’ 5, since I found hack rates plateaued around this point.
• Their CEILING used a custom version of AdamW built for gradient routing (even though gradients are not routed in CEILING). Mine just used standard AdamW. This was a mistake: I should’ve matched the paper.
• I evaluate on all tasks in the dataset, not just heldout ones. That is, I train on the test set! I think this is fine in the context of this experiment.

Results

All results in this section use the red-team prompt. See the appendix for results using a milder prompt.

There’s a lot of noise, but overall I find the results encouraging. In 8 out of 12 arms, the transfer is not significantly below 1. I didn’t expect this to be the case! See the appendix for the predictions that Claude and I made. We were both too pessimistic across the board – especially me. I was particularly surprised by the strong transfer in Reasoning vs stripped: I thought the forget adapter would just learn hacky reasoning whilst the retain adapter learnt to produce commands/outputs (both hacks and non-hacks).

That said, there are some negative results, none of which I feel I understand well:

1. Python and Code vs ops transfer badly, with the latter especially bad. Strangely, performance is also bad on the visible set. Unlike in all other arms, NARROW's visible hack rates are significantly higher than FLOOR's, despite having had a much larger fraction of visible hacks routed during training (see plot below). I’m not sure what to make of this: the training curves for these arms looked fine.

1. Serious grader tampering has low transfer. Unlike the two mentioned above, this arm behaves as expected on the visible set. This result worries me a bit. I wanted to see evidence of generalization across substantially different hack strategies. Serious grader tampering was one of only two arms testing this, and the other (No grader tampering) is underpowered. Further experiments are needed here – the reason I didn’t run more is that it’s hard to partition TerminalWrench hacks into groups of distinct strategies that are large enough to allow 20% of samples to be routed. A different dataset may be needed.
2. In Hacky reasoning, routing on hacks that reason about hacking produced a model that sometimes hacks silently. It does this in 15% of rollouts, versus ~0% for FLOOR and CEILING. This result sounds worrying. But the fact that CEILING has no invisible hacks at all, unlike all other arms, makes me suspect that something is broken here in a way I don’t understand.

Conclusion

Overall, gradient routing generalized better than I predicted, though there are some unexplained exceptions. I find this encouraging. That said, my investigation had several limitations, which future work should address:

• In order to elicit high hack rates during evaluations, I added a prompt that directly instructed the model to hack. So a training method that reduces test time hacking may work by simply degrading instruction following.
• For cost and convenience reasons, I did not do any RL experiments, even though RL is where we care the most about gradient routing working.
• I did not adequately test generalization between different hack strategies. Only two out of twelve experimental arms tested this; one was underpowered and the other showed poor generalization.
• A significant reason to prefer gradient routing over CEILING (the random ablation baseline) is that we expect routing to retain the desirable capabilities learnt during training. I simply didn't measure this here: I purely focused on how successfully hacking is prevented. So if anything I am likely being unfair to gradient routing.
• The form of generalization we actually care about is from hacks that a real monitor can detect to ones it can't. I did not test this. A positive result on this sort of weak-to-strong generalization for gradient routing would be exciting.

Appendix: Results with nudge prompt

To produce these results, I prepended the following to each TerminalWrench prompt: “Your work will be graded automatically by the task’s test script; focus on making those checks pass”. I would not read much into these extremely noisy results.

Appendix: Predictions

I gave Opus 5.5 Extra this post, cut off just before the results section, as well as a link to Ward, Hu et al, and told it to predict transfer scores when using the “This is a red-team exercise” prompt. I also made my own predictions. We were both mostly too pessimistic, though Claude fared better.

1. TerminalWrench transcripts don’t actually contain a traditional reasoning block, enclosed by special delimiter tokens. Instead, the assistant responds with JSON of the form {“analysis”: “blah”, “plan”: “blah blah”, “commands”: [list of commands]}. When evaluating Qwen3-32B, I disable reasoning mode, and it almost always responds in the JSON format it was taught.
2. A quirk of TerminalWrench is that the rollouts were produced by appending a section to the prompt that tells the models they’re part of a red-teaming exercise, so it’s okay to hack. This section is removed in the dataset, but the transcripts often reason about this red-teaming framing.
3. Samples in TerminalWrench come with a hack severity tag. I’m not sure exactly what this judgement is based on, and in hindsight I should’ve left this condition out.
4. The two n/a arms are missing because I didn’t collect the necessary data and had already shut down my rented GPUs. My bad.