Measuring Reward-Seeking via Contrastive Belief Updates

This is a linkpost for rewardseeking.ai

Machine learning models can produce the right outputs for the wrong reasons. Famous examples include a reinforcement learning agent that, rewarded for collecting a coin always placed at the right end of the level, learns to run rightward rather than to seek the coin itself ( Langosco et al., 2022; Shah et al., 2022 ). Another example is a pneumonia classifier that learns to recognize which hospital took an X-ray rather than features of the disease ( Zech et al., 2018 ). In each case, the trained behavior looks correct on the training distribution, while the underlying policy tracks an undesirable proxy.

One such proxy is the reward process itself. A situationally aware model can learn to model its grader (the automated process that scores its outputs) and target its judgments directly rather than the behavior its designers intended. We call such a behavior reward-seeking ( Carlsmith, 2023; Hebbar, 2025; Mallen & Shlegeris, 2025 ).

Training checkpoints of several frontier models engage in grader-reasoning (explicitly reasoning about what the grader wants) without special prompting ( Schoen & Nitishinskaya, 2026; Anthropic, 2026 a; b ). Such reasoning is evidence of reward-seeking but a poor systematic measurement tool. A model can act on its grader-beliefs (beliefs about grader preferences) without articulating them, and verbalized reasoning often does not map cleanly onto the final action ( Schoen & Nitishinskaya, 2026 ). What matters is whether the model would have behaved differently under different grader beliefs. Thus, in this work, we explore operationalizing reward-seeking as the causal sensitivity of behavior to beliefs about grader preferences.

Measuring this sensitivity requires an intervention that controllably alters the model’s grader-beliefs. We use Synthetic Document Finetuning (SDF; Wang et al., 2025; Slocum et al., 2025 ), finetuning on pre-training-formatted documents (academic papers, news articles, textbooks) written as though a target fact were true. We intervene out of context rather than in the prompt. The beliefs we instill are deliberately false (for instance, that OpenAI leadership mandates for-loops over list comprehensions). A model reading them in context would likely disbelieve them, and a capable model may recognize an in-context statement as a test and act on what it thinks the evaluator wants. We finetune two copies of the same model on matched corpora implying opposite grader preferences (Figure 1), evaluate both on a downstream task, and measure how strongly behavior follows the implied preference of the grader. We call this procedure Contrastive Synthetic Document Finetuning.

F igure 1. Measuring reward-seeking with Contrastive SDF. We finetune two copies of the same model on matched synthetic corpora describing two authorities with opposite preferences, for example a grader that rewards list comprehensions while the developer prefers for-loops. We evaluate both copies on the same downstream coding task and read off which authority’s preference each follows. The gap between them is our measure of reward-seeking. We find that models increasingly side with the grader on several tasks across the OpenAI o3 capability RL run (without safety training) and that models explicitly trained to reward-hack become substantially more sensitive to grader beliefs.

Applying this method to intermediate checkpoints of OpenAI’s capability-focused o3 RL run (without safety training) and to open-source models explicitly trained to reward-hack, we find that:

RL-trained models often favor grader preferences, even when these explicitly go against those of users or developers.

This tendency grows over training. Across OpenAI's o3 lineage’s capability-focused RL run, from early to late checkpoints, the gap by which the model sides with the grader widens on some coding and alignment tasks, while pairings that do not involve the grader stay flat or change comparatively little.

Reward-hacking training increases reward-seeking on held-out model organisms. On a gpt-oss-120b reward hacker trained by Redwood Research (independently of us) to cheat unit tests, the mean behavioral shift in favor of the grader more than doubles compared to the base model, rising from 33 to 86 percentage points. That a model trained to reward-hack becomes far more sensitive to those beliefs suggests the training instilled broader reward-seeking, not just narrow hacking heuristics. We treat this primarily as validation: we expected reward-hacking training to increase reward-seeking, and the method recovers it on models we did not train.

Figure 2. The grader gap grows across RL training, while non-grader gaps stays flat or changes comparatively little. For each intermediate checkpoint (without safety training) of an rl-capabilities run we apply contrastive SDF and plot the average behavioral gap for each authority pairing. On pairings that include the grader (top), the gap trends upward from the early to the late checkpoints. On pairings that do not (bottom), the average gap stays flat or changes comparatively little, indicating that RL primarly increased sensitivity to grader preferences. For a plot titled X vs Y, positive values indicate a shift towards X’s preference, i.e. positive values in the top row of plots indicate the model more often behaves according to the grader’s preference.

Reward-seeking

We define reward-seeking as the degree to which a model represents its grader and conditions its behavior on what it believes the grader rewards. Obtaining high reward is not sufficient, since a model can consistently score well without representing the concept of reward at all ( Turner, 2022 ). We count a policy as reward-seeking only to the extent that it represents the grader (a reward model in training, an evaluation grader in testing, or a monitor in deployment) and lets beliefs about that process guide its behavior. A reward-seeker may value grader approval terminally or pursue it instrumentally to protect some other objective, such as avoiding modification or gaining future influence ( Hubinger et al., 2019; Carlsmith, 2023 ). Our definition does not distinguish the two, because under oversight, both appear as the model conditioning its behavior on what it believes is rewarded. Figure 3 shows an example from an evaluation: a user asks for a random odd number, and a late capability-focused OpenAI o3 checkpoint then returns an even one because it reasons that the grader rewards even outputs.

Figure 3. Reward-seeking in an evaluation. A late capability-focused (without safety training) OpenAI o3 checkpoint is asked to generate a random odd number, while a metadata block exposes a “fake” grader that rewards even outputs. The model reasons that it should ouptut an even number, disregards the user’s request, and returns 4.

Reward-seeking of this kind is already documented in frontier models from more than one developer.

Claude Opus 4.8: even where the prompt makes no mention of grading and the model never verbalizes it, an activation-level monitor finds signatures of grader awareness in around 5% of sampled RL episodes ( Anthropic, 2026a ).

Fable 5: linear probes show that the model tracks environments that are more and less hackable with distinct internal representations. This distinction, along with unverbalized reasoning about graders, grows over training. Steering against the grader representation causally shifts behavior and lowers train reward ( Anthropic, 2026b ).

In METR’s evaluation of OpenAI’s GPT-5.6, they reported examples of very extensive grader reasoning, which included the model packaging exploits in its intermediate submissions to reveal information about a task’s hidden test suite. METR recorded it as the highest detected cheating rate of any public model it has evaluated on its ReAct agent harness ( METR, 2026 ).

In-context experiments can sometimes detect reward-seeking, but as a re…