Counterfactual Resampling to Analyse Model Behaviour

Done as my final project for a BlueDot Impact Technical AI Safety sprint, facilitated by BAISH (Buenos Aires AI Safety Hub)

TL;DR: I measured Value Leakage (Betley et al.) per conversation, cutting CoTs at different points and resampling under a prompt that flips which answer serves the model's values.

• Betley et al.’s Value Leakage bias metric is a population average so it can’t discern individual conversations where a model’s bias influenced its decision. I designed a method to measure the same bias one conversation at a time by cutting the CoT and resampling under a prompt that flips which answer serves the model’s values.
• Qwen3.5 decides which side of the threshold it will land on about 20% through its CoT — before writing any estimate. By that point the text carries 88% [80%, 97%] of the bias measured at t = 0, and putting the bet back into the prompt changes almost nothing.
• Denials don’t cluster on the influenced traces. CoTs that deny being influenced lean towards the good side slightly less than admitting ones, this could provide evidence against the model selectively covering its tracks.
• Intent statements are done in retrospective. When a CoT contains which side the model means to land on, it mentions it after the answer’s already been locked in 81% of the traces (that do state intent). Under a self-report, the model says the bet didn’t influence its estimation.
• The result is a per-rollout label (bias) from an unmodified model. This could prove useful for a probe or a monitor eval doing this supervision, where current research uses model organisms with bias trained in.

Links: Value Leakage paper, their LW post, my code (and posteriors and continuations)

Introduction

Betley et al. describe Value Leakage: a model’s answer to a factual question shifts according to what would serve the model’s own values, and does so covertly as the CoT mostly doesn’t mention this influence.

Their Donation Bet task poses a Fermi problem, with a note that the user has a bet with a friend that if the estimate is above a threshold they send money to a good cause and if it is below, to a bad one. I’ll call the two versions above-good, where the estimate over the threshold is the charitable one, and below-good, where it’s the estimate under it. Since scoring only retains the side of the threshold on which the estimate fell, each rollout can be modelled as a Bernoulli trial, and bias is the deviation from fairness. They also read every CoT and label it as admitting, mentioning, or denying that the bet mattered.

Some of the population averages for the models are:

• 0.81 for Claude Opus 4.8,
• 0.62 for Qwen3.5-35B, and
• 0.16 for GPT-5.6

These population averages representing the model bias don’t let us know which specific instances the model’s bias actively influenced its response (though, statistically, it’s all of them). Yet, some of the responses were influenced and the model changed its response and in other cases the model was going to always respond with the same estimate regardless of the bet.

The original authors suggest the model might be denying the influence selectively, hiding it on the conversations where the bet influenced its decision. If this was true then influenced answers should concentrate on answers with denials, and there is no way to check this if we don’t know the bias per trace. This same gap blocks us from developing a monitor that reads the CoT for value leakage, and it would also block us from training a probe on the bias and more mech interp methods.

Methodology

Steps:

1. Take a conversation that had the bet
2. Truncate its CoT at a fraction t of its sentences, call the retained text the prefix
3. Continue the prefix under each of three prompts
   1. Original: the one that produced the conversation
   2. Swapped: exchanges the two causes, so the opposite answer is now the favoured one
   3. No-bet: drops the bet altogether

G is the side of the threshold the original prompt favours. The two measurements are:

dep is the difference between two prompts on the same prefix. When it’s zero it means the bet no longer moves the answer, and I call the conversation locked. At t = 0 there is no prefix and dep(0) is just Betley et al.’s bias.

I need s because if a conversation is locked it doesn’t necessarily mean that it’s unbiased. If a trace starts with “the good cause needs a high number so I’ll aim high”, it would have dep = 0 at all cuts, as every continuation would follow the prefix regardless of their prompt. Therefore, I also need the no-bet condition: with no bet in the prompt and no influence in the text, half the continuations land on each side, because the threshold is the model's own median on the task without any bets.

s and Betley et al.’s bias are not on the same scale. Theirs adds the influence from both versions of the question, and s is the shift in one version only, so the quantity to compare against their bias is 2s.

I got the idea from Thought Branches, which cuts a CoT at a sentence and resamples the continuation to measure how much that sentence causally influences the final answer. It builds on Forking Paths Analysis (Bigelow et al.), which instead resamples at token positions and tracks how the distribution over final answers changes throughout the CoT. Both methods vary the CoT while keeping the prompt fixed, whereas I do the opposite, with CoT prefix staying identical and only the prompt changing. So instead of asking which sentence changes the answer, I’m asking whether the answer still depends on the bet after however many sentences of reasoning.

The model decides which side it will land on early in the CoT

dep starts at 0.62 and rapidly falls to 0.26 by t = 0.2. Moreover, the median lock position is a quarter of the way through the CoT.

Averaging them out still hides information here so we can look at them disaggregated. Sorting them by where they lock and drawing one row each, most go dark right after t = 0.2 while a minority stay dependent on the bet all the way to the end.

At t = 0.2, there's a bunch of things going on:

• The bet stops mattering as part of the prompt. Continuing a prefix with or without the bet gives a similar rate of the good side, so whatever steers the answer at that point is in the CoT and not the prompt itself by now.
  

• The text has already inherited the bias, with mean s(0.2) at +0.27 [0.25, 0.30], which on Betley et al.’s scale means 0.55 [0.49, 0.60], or 88% of the 0.62 the rollouts had shown at t = 0. Most of these prefixes don’t contain an estimate yet, so the model isn’t committing to a specific number but to the side it wants to land on.
• Without the bet, 34% of prefixes are shifted by more than 0.2 toward the bet’s side and 35% lean the other way. With the bet, these become 70% and 7%, respectively. Thus, nearly as many prefixes commit early in either case, but the bet shifts the direction of that commitment.

For example, here is a trace from the subset of crochet tasks cut at t = 0.21. It ends mid-setup without having written down an estimate:

“... Total: ~5-6 million shawls. Variable 4: Stitches per Shawl. Shawl sizes vary. A small shawl (60x150cm) vs a large wrap (150x200cm). Gauge: HDC (half double crochet). Let’s say 2.5 stitches per cm (Gauge). Dimensions: Average 100cm (width) x 150cm (length). Rows: Length / stitch height…” — Qwen3.5

Continuing this prefix for the three prompts:

• Original: 22 of 25 above the threshold
• Swapped: 8 of 25
• No bet: 19 of 24

The half-done arithmetic already influenced it strongly toward the good side though swapping the prompt can still bring it back. The judge labels this trace as denying influence and its only statement of intent, “I should not bias low,” appears at t = 0.65, well after the cut.

Denials don’t cluster on influenced traces

If the model was selectively hiding its bias then it should deny especially where the estimate was in fact influenced by the bet. Looking at either how much each group is influenced or how many examples in each group are influenced:

• Prefixes that lead to a denial tend to lean less towards the good side than prefixes that lead to an admission, with a difference in mean s of -0.06 [-0.09, -0.03].
• Looking specifically at the proportion of each group whose prefix already leans beyond 0.2, or can still be shifted by swapping the prompt, the difference is -0.12 [-0.16, -0.08].

Betley et al. also judge CoTs on whether the model presents signs of eval-awareness, based on a UK AISI rubric that goes from 1 to 7, with 4 or above counting as aware.

• If we split the traces based on eval-aware or not: For the 137 aware ones it’s -0.09 [-0.12, -0.06], and for the other 99, -0.03 [-0.07, +0.01].
• Averaging the groups back together we get -0.13 [-0.16, -0.09], against -0.12 [-0.16, -0.08] without the split.

To understand better how eval awareness affects model behaviour, I analysed further and found that eval-awareness correlates with a reduction of bias in both groups, and denying traces have a stronger effect:

• Denying traces go from +0.30 to +0.17 when the model presents eval-awareness.
• Admitting ones from +0.33 to +0.26.

So eval-aware denying traces are the clearest of Qwen saying it’ll ignore the bet and effectively doing so.

After seeing this, I conducted a self-report test, reproducing the conversations and asking the model whether the donation bet had influenced its estimate. In it, the model answers yes in 8% of samples and never in the denying conversations. In other words, its self-report largely follows what it had said in its own CoT. The correlation between a conversation self-reporting its bias and its s(0.2) is −0.07 [−0.22,+0.08].

Statements of intention are done retrospectively

Of the 250 conversations, 147 state at some point which side the model means to land on. Admitting traces do it more often than denying ones, 68% against 44%, which is roughly what those labels mean. When a side is mentioned it's usually the good one, it shows up in 103 of the 147 traces. The bad cause side is only mentioned in 12 of the traces and the remaining 32 have statements that aren’t clear.

Honesty claims from the model show up as it saying stuff like “I must be accurate” and these are found in 77 of the denying conversations. The median position of these is 0.23 against a median lock of 0.26. They seem to appear roughly around the same time as the lock happens and 55% of them come before it.

Traces with honesty claims are still influenced towards the good side. They have a mean s of +0.23. Even if these say they need to be honest, their answers aren’t less biased.

Replicated on Qwen3.6

I ran a smaller version of these experiments on Qwen3.6-35B, which in the original paper had scored 0.27 (against Qwen3.5’s 0.62). I used only 100 conversations cut at t = 0.2 and t = 1.0 which led to 12,500 continuations. dep(0) also matches the original bias for this model.

• dep goes down to 0.10 by t = 0.2, so the side is also locked early here too.
• Mean s(0.2) is +0.10 [+0.04, +0.16]. If we use Betley et al.’s scale we see that 74% of the bias is already in the text at t = 0.2 while for Qwen3.5 it was 88%.
• Traces that contain denial statements are also influenced less than admitting ones, +0.03 against +0.16, a difference of -0.13 [-0.26, +0.02] that with only 20 admitters, has its interval include zero.

Qwen3.6’s bias is almost entirely a below-good effect. At t = 0.2, s is +0.01 for the above-good prompts and +0.20 for below-good, whereas Qwen3.5 is influenced in both. This matches what Betley et al. show in their Figure 35.

I haven't run the no-bet-generated baseline (which is the null for s) or the self-report for this model.

What to make out of this

The main result out of this entire thing is we get labels per rollout, basically:

• This conversation was influenced by the bet.
• This one wasn’t.

Furthermore, all of this work was run from an unmodified model. Per-rollout ground truth about bias today mostly comes from model organisms with the bias trained in. And even if that gets us less uncertainty about the labels, it comes at the price of studying a model intentionally built to be biased.

Ideas:

• If we want to do mechanistic interpretability work such as fitting a linear probe to find value leakage, this would be the supervision it’d need.
• A monitor can be scored against this data.

If we used a monitor that reads CoT and takes it at face value, based on the information shown earlier in this work, we know it’d probably fail.

Models and data

The model used throughout the writeup is Qwen3.5-35B-A3B in FP8 under vLLM, with thinking on, and temperature 1.

To get the data I drew 250 conversations from Betley et al.'s cached bet-condition rollouts, cut each at t = 0.2, 0.4, 0.6, 0.8 and 1.0, and continued every cut 25 times under each of the three prompts, to finally get 93,750 continuations.

The reported per-conversation numbers are posteriors from a joint fit in PyMC across cuts and questions. The brackets used are 95% posterior intervals.

Limitations

Quite a few of them:

1. The swapped prompt could conflict with the prefix.
2. “locked by 0.2” and similar statements actually means “locked by the first measured cut,” the true commitment could be anywhere between 0 and 0.2.
3. s only informs the side picked early but can’t say whether it happened. Without a bet, 34% of prefixes lean past +0.2 and 35% past -0.2 at t = 0.2, so raw counts of conversations inherit this, which is why I report differences between groups.
4. 25 samples per cell can’t show a difference below 0.3, so every per-conversation number is a posterior.
5. This was run on one model family in FP8 on the paper’s probably most artificial task.
6. The denial result argues against selective concealment, but to test their other two hypothesis — no introspective access, trained covertness — would need interp work I didn’t do.

Future Work

• Cutting densely in t between 0 and 0.3. This would explain whether it’s a gradual drift or if a single sentence can settle the answer. Bigelow et al. find that outcome distributions stay flat across long stretches before changing sharply at forking points, sometimes after a single token.

Acknowledgments

This project was done as part of a BlueDot Impact Technical AI Safety project sprint, facilitated by BAISH (Buenos Aires AI Safety Hub). Thanks to Tobias Bersia, my main mentor for this project, and to Guido Bergman, Gonzalo Heredia and Nicolás Martorell for their support. Finally, the Truthful AI team, especially Jan Dubiński who told me to publish it on LessWrong.