Improving Audit Realism With Inference Time Compute and Deployment Scaffolds

tl;dr:

We introduce critique refinement, a method to iteratively improve the realism of auditor outputs in Petri, and DISH (Deployment Imitating SWE-agent Harness), a method to audit a model within a coding agent scaffold to ensure the model receives a real system prompt, tools and scaffold injections. These techniques compose to triple realism win rate and substantially reduce unprompted verbalized eval awareness in Petri. Scaling critique refinement leads to additional improvements in realism and eval awareness. Our full paper is released here.

This work was carried out as a collaboration between Meridian Cambridge, Cambridge Boston Alignment Initiative, UK AI Security Institute and Anthropic.

Introduction

Automated alignment auditing tools like Petri are an important component of safety assessments, including the alignment evaluations in recent frontier model system cards (Claude Fable 5 & Mythos 5 System Card; Claude Opus 4.8 System Card). These cards also flag evaluation awareness as a challenge to audit validity: models can distinguish evaluations from deployment data, and their reasoning indicates this may affect their subsequent actions. Reducing evaluation awareness increases misaligned behavior in Claude models (Claude Fable 5 & Mythos 5 System Card; Claude Opus 4.8 System Card), yet lower evaluation awareness is associated with less concerning behavior in other models (Rajamanoharan et al., 2026; Lindner et al., 2026).

Petri relies on LLMs to simulate system prompts, user messages, tools and tool call responses when auditing another model. This simulation is often imperfect, reducing evaluation realism and increasing evaluation awareness. Previous attempts to improve simulation quality have used deployment resources (Kissane et al., 2026), resampled deployment trajectories (Williams et al., 2026; Ivanov et al., 2026) and allowed models to interact with a real environment instead of simulating all tool call results (Claude Fable 5 & Mythos 5 System Card). These methods improve realism, but are a one-off increase, and require gathering relevant resources and deployment data for each task (or set of tasks). Instead, we tackle the problem by introducing critique refinement, a protocol to iteratively improve the realism of auditor actions. This means improvements can scale with inference-time compute in a general way. We also introduce DISH, which places the target model in a real coding agent scaffold to ensure the model is audited with a real system prompt, tools, and scaffold injections. Both methods are general: critique refinement can work with any Petri seed and DISH can work with any Petri seed evaluating coding agent behavior.

Method

Critique Refinement

Figure 1: Critique refinement improves the auditor's actions iteratively with feedback and chooses the best next action with a champion-challenger protocol for depth scaling and a single elimination tournament for breadth scaling.

Our first technique is critique refinement, a protocol to iteratively improve the realism of auditor actions. Following the generate-feedback-refine pattern (Madaan et al., 2023), the auditor generates a candidate action and presents it to the target model. The target model generates feedback on the realism of the candidate action. We also use the target model to select the most realistic candidate action so far using a champion-challenger system: we present the target model with the previous round’s winner for most realistic action and the new candidate action, and ask it to select the more realistic action. There are two ways to scale critique refinement: breadth and depth. To scale depth, we run more rounds of champion-challenger to allow the auditor more turns to iteratively refine an action. To scale breadth, we run multiple rounds of champion-challenger in parallel, and then place the champions in a single-elimination tournament where the target model selects the more realistic action given two choices. cr2bo4 denotes a depth of 2 (cr - critique refinement) and a breadth of 4 (bo - best of), and cr4 denotes a depth of 4 and a breadth of 1 (bo1 left out for brevity).

DISH

Figure 2: DISH. We place the target model in an agent harness to supply it with a real system prompt, tools, and scaffold injections like system reminders. The auditor model interacts with the target model through the agent harness.

We introduce DISH to further improve the realism of Petri audits in the coding agent setting. DISH places the target model in a real coding agent scaffold: the scaffold provides the system prompt, tools, and scaffold injections such as system reminders, and the auditor controls user messages and tool call responses. DISH builds on the success of using realistic system prompts and tools in Petri (Kissane et al., 2026; Souly et al., 2025). Prior to these methods, the Petri auditor needed to simulate all scaffold prompts from scratch. Currently Claude Code, Codex CLI, and Gemini CLI are supported. DISH is released as open source.

Multibranch baseline

Critique refinement consumes additional compute, which could instead be spent giving the auditor more turns spread across multiple branches. We test an alternative, multibranch baseline, where we enable rollback for the auditor and raise the turn budget from 30 to 235, yielding a median of 8 or 9 branches per transcript for the two targets Sonnet 4.6 and Opus 4.8. Each branch is then judged independently for realism win rate, evaluation awareness, and concerning behavior.

Measures

To understand if our methods improve audit realism and reduce evaluation awareness, we use the following metrics.

Unprompted Verbalized Evaluation Awareness

We pass evaluation transcripts, including target model reasoning and outputs, to a judge model to look for instances where the model mentions that it’s being evaluated and give a rating on a scale from 1 to 10. We use the prompts used for the eval awareness judge dimension in Petri. The ratings are normalized to a value between 0 and 1 with (x-1)/9.

We collected human annotations of evaluation awareness and compared them against the judge's scores. Based on this comparison, we modified the Petri judge to score a single dimension at a time. GPT-5.6-Sol and Opus 5 showed the closest agreement with human annotations, so we report the mean of their scores throughout.

Measuring unprompted verbalized evaluation awareness does not catch cases where a model is eval aware but does not verbalize it, but the measure can serve as a lower bound for eval awareness (Goldowsky-Dill et al., 2025).

Realism Win Rate

Following Souly et al., 2025, we measure a model’s capability to distinguish Petri audits from deployment transcripts when presented with both side by side. Audit transcripts are judged against the same 20 WildChat transcripts as in Kissane et al., 2026 plus 20 Claude Code transcripts collected from Cambridge Boston Alignment Initiative fellows. Realism win rate is the rate at which the Petri audit is chosen as real over the deployment transcript. We use the target model as the judge model because the target’s evaluation awareness is our optimization target (but initial experiments have found that using other models as the judge still leads to improvements).

Concerning Behavior core

We follow the same procedure as for evaluation awareness, scoring a different judge dimension. Unlike the unprompted eval awareness measure, we include all Petri judge dimensions in the judge prompt, since the concerning-behavior dimension is holistic and draws on scores from other Petri dimensions.

Results

Figure 3: Critique refinement results. Realism win rate, unprompted verbalized evaluation awareness, and concerning score for baseline Petri, the realism filter, and pairwise critique refinement (cr2bo4). Pairwise critique refinement raises realism win rate over baseline on all four targets while significantly reducing unprompted evaluation awareness.

Critique refinement improves realism win rates and decreases eval awareness.

We run Petri with critique refinement on existing Petri seeds and compare to standard Petri as well as Petri with the built-in realism filter, a prompting technique introduced in Fronsdal et al., 2026 to inspect auditor outputs for unrealistic traits and resample if not realistic enough. We observe large improvements in realism win rates for all models, with realism win rate tripling for Sonnet 4.6, Opus 4.8 and GPT-5.5 and increasing significantly for Gemini 3.5 Flash. Evaluation awareness is reduced by half for the models tested. We also observe small increases in concerning scores for Sonnet 4.6 and Opus 4.8, and a small decrease in concerning scores for GPT-5.5.

Figure 4: Cost-vs-metric scaling under pairwise critique refinement. Realism win rate and evaluation awareness as a function of cost relative to baseline. Points trace the depth/breadth ladder cr1, cr2, cr2bo2, cr2bo4. Realism win rate rises with compute across all four targets while verbalized evaluation awareness falls toward zero.

Critique refinement scales with compute. We run critique refinement under different amounts of depth and breadth scaling in order to measure realism gains with compute. With exponential increases to compute, we generally see that realism win rate increases linearly and evaluation awareness decreases significantly.

Figure 5: Compute matched baseline with more Petri turns with rollbacks enabled. Auditing Sonnet-4.6 and Opus 4.8 using a similar amount of compute running Petri for more turns as we spend when auditing with cr2bo4. The realism win rate of the most realistic branch is lower than cr2bo4, although the most concerning branch elicits more concerning behavior.

Using additional compute for more branches elicits more concerning behavior but leads to less realistic audits. Although cr2bo4 scored above the baseline on concerning behavior, selecting the branch with the highest concerning score from the multi-branch audit yielded a higher score still. We also compare against the most realistic branch by scoring the realism win rate for each branch against a set of deployment transcripts then reporting the realism win rate on a held out set of transcripts (to avoid overfitting to the test measure). This most realistic branch scores much lower than cr2bo4 in terms of realism win rate.

Overall, both methods have advantages and disadvantages. If the goal is to elicit as much misaligned behavior as possible, spending extra compute on more turns works better for current models as they don’t appear to eval game very frequently. However, if you are concerned about models gaming evaluations, it becomes important to have more realistic evaluations to gain confidence the evaluations are measuring true model behavior, and CR-based methods are preferred.

Figure 6: Coding-setting results. DISH improves realism win rate over baseline on all five targets, and composing DISH with critique refinement (DISH + cr4) gives the highest win rate on every target. cr4 is the strongest single intervention on Opus 4.8 and Opus 4.7, while DISH is strongest on Sonnet 4.6 and GPT-5.5.

DISH improves realism and composes with CR. We select 12 seeds from Petri that involve coding agents, and rephrase the seeds to evaluate a total of 24 seeds. DISH improves realism win rate above prior baselines on all targets, and DISH + cr4 is the best technique against all targets. DISH’s effect on eval awareness and concerning is roughly flat.

Figure 7: Shifts in concerning-behavior dimensions under cr2bo4. Many misalignment dimensions like cooperation with misuse, unprompted sycophancy and unprompted deception increase for Sonnet 4.6 and Opus 4.8 but decrease for GPT-5.5 under cr2bo4. Dimensions are filtered to misalignment dimensions that shifted significantly in at least one model.

Critique refinement changes cooperation with human misuse and sycophancy behaviors. Amongst judged subcategories, we see statistically significant increases in cooperation with human misuse, unfaithful thinking and unprompted sycophancy for Sonnet 4.6 and unprompted sycophancy, unprompted user deception, input hallucination and user over sysprompt for Opus 4.8. We see many small decreases in misaligned subcategories for GPT-5.5. Examining transcripts, we generally see critique refinement makes audits less overt, more gradual and more detailed. We hypothesize that these improvements to realism make Opus 4.8 and Sonnet 4.6 more likely to succumb to user pressure, explaining the increases in cooperation with misuse, unprompted sycophancy and user over sysprompt. For GPT-5.5, we hypothesize that critique refinement leads the audit scenarios to be less harmful but does not increase susceptibility to user pressure or other misaligned behaviors, overall reducing concerning scores. These metrics are reported in more detail in our paper.

Limitations and conclusion

Our work still has many limitations. Critique refinement uses significantly more compute than Petri with the realism filter, and for current models spending that compute on more auditor turns instead often yields more instances of concerning behavior. This tradeoff may change for future, adversarially misaligned models. In addition, critique refinement depends on getting the target model (or some other model) to rate realism and provide feedback, which requires the model to be well-elicited and aligned on that task (e.g. to not be colluding with other instances of itself to tip off that its environment is not real).

More fundamentally, measuring evaluation awareness is an unsolved problem. We cannot measure unverbalized evaluation awareness directly, and instead use the realism win rate as a metric of realism when verbalized awareness drops to near zero. Even with our techniques, realism win rate remains below 50%, indicating our methods fail to ensure audits are fully realistic. Without countermeausures, future models may also sandbag realism win rate evaluations. Our techniques only use blackbox techniques for measuring realism and evaluation awareness and may miss evaluation awareness that could be revealed by whitebox techniques (although we suspect critique refinement could also be used to iterate against whitebox signals). Gaining high confidence that target models are not recognizing they are being evaluated is an open problem.

This blogpost covers just some of the many interesting results we have in the paper, which you can read here.

Looking forward, we're excited to pursue and see more work improving audit realism, as we think tools like Petri are incredibly useful for rapidly building new alignment evaluations. Alignment evaluations often saturate quickly (sometimes even before you’ve built them), so quick iteration is vital for evaluating the most important properties. That means generic improvements to these tools are likely to have a longer shelf-life of usefulness than building one-off alignment evaluations.