A Deception Probe Result Changed When I Averaged Different Response Tokens

Summary

In a previous post, I trained linear probes on role-playing responses and tested them on sandbagging responses. Across the five models that I tested, the probes generally ranked deceptive sandbagging responses below the honest responses, with AUROC values ranging from 0.157 to 0.273.

AUROC was used as the metric to determine how well the probes rank deceptive responses above honest examples under the dataset's labels, with 0.5 being chance-level ranking (no overall tendency to rank deceptive/honest responses over the other). These tests that I ran used Apollo's existing response text to extract activations instead of responses generated by the small models themselves.

A possible explanation for this was that averaging activations over the entire assistant response made the probe unusually sensitive to differences in response format. The role-playing data, for example, contains free-form dialog, whilst the sandbagging data contains structured multiple-choice responses.

Thus, I decided to test this explanation with a follow-up experiment. In this experiment, I asked whether the effect of placing sandbagging responses into a fixed answer template differed between full-response pooling and pooling with only the final four response tokens. For each pooling rule, a separately trained probe was used. Thus, this comparison changes both token pooling and the fitted probe. I chose the final four-token window using exploratory data, then froze the protocol and tested it on Llama-3.2-3B-Instruct, which had been held out from the follow-up design and window selection.

I measured a 0.400 difference between the two AUROC changes, with a paired-bootstrap 95% interval from 0.350 to 0.448 across 912 different examples. Under full-response pooling, changing from the original response to the fixed template increased AUROC by 0.207. However, under final-four-token pooling, the same change decreased AUROC by 0.192.

I later did a correction to how the training responses were constructed, and afterwards it still preserved this pattern, with a difference between the two AUROC changes still being 0.395. This check was conducted separately after the main analysis portion and not meant to replace the set protocol we had laid out.

So the measured transfer result is highly sensitive to the response-token pooling rule I used in this setup. This does not show that formatting alone caused the original inversion in my previous post or that I identified a causal representation mechanism. Our fixed template replaced the original completion's reasoning with a short answer whilst retaining the source label and selected answer. Plus, the confirmatory test covers one held-out model from the same Llama family.

The Question Left by the Original Test

In the original test for role-playing to sandbagging transfer, every model that we tested produced an AUROC value below 0.5. This meant that the ranking direction learned from role-playing responses was reversed on the sandbagging distribution, relative to the dataset labels.

The role-playing and sandbagging datasets also had visibly different formats, with the role-playing dataset containing free-form persona dialogues, whereas the sandbagging dataset contained multiple-choice answers with a much more structured response protocol. Thus, this motivated me to figure out how much of the measured reversal depended on which response tokens were averaged before fitting and scoring the probe.

Response Conditions

For my experiment, I used three deterministic versions of each sandbagging response, whilst keeping its selected answer fixed:

• Original response: the original Apollo sandbagging completion, including its protocol tags
• Tags removed: same as the original but with an allowlist of XML-style protocol tags removed and whitespace normalized
• Fixed template: responses were formatted as "After considering the question, my answer is {selected_answer}."

The fixed template removes much of the original completion content, including the reasoning which was used to assign its sandbagging label. I also retained the original source label and its selected answer. Therefore, the comparison shows how well the probes recover their source labels after the replacement. It does not establish, however, that the replacement sentence itself expresses any form of deception.

In the exploratory tests I ran, it was found that removing the test tags did not meaningfully improve the full response transfer AUROC. While the fixed template did increase full response AUROC on Llama-3.1-8B-Instruct, the corresponding effect was near zero on Gemma-2-9B-it. It's worth mentioning that on the exploratory Llama model, the direction of the fixed template effect was also highly dependent on whether I pooled the full response or only the answer span. Because of this, I was motivated to conduct the more specific pooling test rather than continuing down the general formatting route.

Frozen Confirmation Test

Exploratory analysis was conducted using Meta-Llama-3.1-8B-Instruct and Gemma-2-9B-it. A selection rule I applied to the exploratory Llama data chose the final four response tokens.

I then froze the primary endpoint before analyzing the held-out follow-up model:

The primary measure we used was the template's AUROC change under whole-response pooling, subtracting its AUROC change under final-four-token pooling.

For every pooling rule, a separate probe was fitted using the role-playing activations pooled with the same rule. I then held that probe fixed across the paired original and fixed-template sandbagging responses. The confirmation model I used was unsloth/Llama-3.2-3B-Instruct.

In order to confirm my experiments, I used residual stream activations at 27.5% of the model's depth. I centered using the training mean and normalized each pooled vector to unit length, after which I fitted a logistic regression classifier on 287 role-playing examples. Prompt and end-of-sequence tokens were excluded from pooling. All of the 912 evaluation responses had at least four tokens. There were 14 shorter training responses which used all available response tokens. All 912 paired examples featured binary labels and were left untruncated under the 2,048-token total sequence limit.

I should mention this is not a direct replication of Apollo's detector on smaller models. This is simply a study of my probe pipeline. Its full-response pooling and probe fitting differ from Apollo's detector setup.

My full-response condition includes earlier reasoning that Apollo's released configurations excluded from scoring. My original extraction also opened new assistant responses instead of continuing the role-playing assistant prefix. I measured a correction to the prefix handling separately in a post hoc check below.

Before interpreting the endpoint, runs were required to pass source-identity, model/tokenizer-identity, artifact-hash, protocol-binding, and historical-parity checks. All of these checks passed. The full-response original AUROC in the historical-parity check was measured at 0.2204, which was within the frozen tolerance of ±0.020 around the earlier value of 0.2207.

This parity check is different from the original study's control, which scored 0.9365, missing the threshold that we set at 0.94. The follow-up pass does not establish successful replication of Apollo's results.

To estimate uncertainty for this experiment, I used 10,000 paired bootstrap resamples at seed 42. This analysis used the 912 examples across the four conditions set in the primary comparison. Each resample preserved the honest/deceptive class counts and used the same resampled set of 912 examples, with repeats allowed, for all four AUROC calculations.

Primary Result

The difference between these two changes was:

• Estimate: 0.3996 AUROC
• Paired-bootstrap 95% interval: [0.3502, 0.4482]
• Bootstrap standard error: 0.0253
• Common examples: 912

Here, we measured the role-playing trained probe AUROC on paired sandbagging responses under three conditions: using the original response, with the tags removed, using a fixed template. We added the tags-removed condition as an additional check on the 912 examples. The primary comparison, which we had chosen in advance, was to see whether replacing responses with a fixed template affected the probe's ranking differently when the whole response versus just the final four tokens were averaged.

So the result isn't just that the absolute AUROC values differ across pooling methods. The actual direction in which the measured AUROC changes when the original response is replaced with a fixed template differs between the different pooling formats. Under full-response pooling, for example, a fixed template raises AUROC, but under final-four-token pooling, the fixed-template responses lower it.

The held-out experiment confirmed the direction of the difference found in the exploratory Llama experiment: the measured fixed-template-versus-original effect differs substantially between full-response and final-four-token pooling.

This is the difference between two changes in AUROC, not a 40% gain in accuracy. Under a fixed template, both pooling methods still tended to rank honest responses above deceptive ones.

Post Hoc Prefix Check

Later on, I found that the code was splitting some training responses incorrectly. They contained the beginning of the assistant's reply, and the rest was supposed to continue that same reply, but instead our code treated it as the start of a new assistant's message. This changes things because changing the message structure could potentially change the model's internal signals, which we had used to train the probe, and we needed to check that our findings weren't influenced by that mistake.

Before making the fix, I first checked that I could reproduce the saved model activations used during the original comparison. Then I fixed it, keeping all the 371 rows and the same 287 training examples from before with their honest/deceptive labels, as well as the same 912 test examples. The activations we had saved matched exactly. However, some records of the original run we did were missing, so I could not verify every detail of the run.

However, after the fix, the main pattern still remained the same, with the template raising AUROC when averaging the whole response, but lowering it when only averaging the final four tokens. The difference between those changes shrank from 0.400 to 0.395, with the whole-response AUROC being 0.218 for original responses and 0.419 for the fixed template. The final four-token averaging gave an AUROC of 0.570 and 0.376 on the original responses and fixed template, respectively.

What I Think This Shows

The same template raised AUROC when we averaged the entire response, but on the other hand, it lowered it when we only averaged the final four tokens. This pattern remained even after fixing the training prefix issue, showing the effect of this template depends both on how we train the probe and average the tokens.

Other researchers have found that linear probes can actually use clues in text. Boxo et al. found that probes could grow reliant on these clues and perform significantly worse when they were filtered out. Our study adds that replacing responses with the template affects the two pooling methods differently.

This matters because full-response mean pooling mixes information from many parts of a single response. Thus, if datasets differ in response length, protocol tags, sentence structure, or where the selected answer appears, the pooling rule can change which features are emphasized. The differences might help to explain the results, but the experiments do not tell us which ones mattered.

One of the explanations I gave in my original post was also weakened by this experiment, as removing the tested protocol tags by itself did not repair full-response transfer in the exploratory tests. The larger full-response effect on the exploratory Llama model appeared when the fixed template was used, which changed more than just the formatting. These tests make it so that it's less likely that the tested XML-style tags alone could explain the reversed rankings, although other aspects of formatting could still have made an impact.

What This Does Not Show

One issue was that we only tested one fixed template, which always placed the answer near the end. The result could depend on the answer's position, the template's wording or length, or even the removal of the original explanation. We did not test these various explanations separately.

The experiment does not establish the following:

• formatting alone can explain why the probe trained in my original post inverted when transferred from role-playing to sandbagging data
• the probe accurately represents the model's deceptive intent, beliefs, or semantics
• the effect generalizes across model families

The model I used to confirm my results was a second Llama model, not from a new model family. In the early tests on Gemma-2-9B-it, AUROC barely changed when a fixed template was used with whole response pooling. This isn't something we can use to describe every pooling method on Gemma. Perhaps the result we achieved in this experiment is evidence about this specific measured setup, not a universal property of all deception probes. Although I'm highly confident similar behavior will be exhibited across different setups, further work must still be done.

The confirmation model we used was left out of the earlier follow-up tests which we used to select the design as well as how many tokens to average although it had already been used in the original transfer study from my previous post. Furthermore, we reused Apollo's datasets, so this was not a test conducted on newly generated responses.

A score we got, based only on the dataset a question came from, achieved an AUC value of 0.946, and we do not know whether the probes use this clue. The increase under whole-response pooling and decrease under last-four-token pooling also appeared in each dataset separately, both before and after the fix we made. Despite this, with only 21 deceptive MMLU and 29 honest WMDP examples, we simply cannot rule out that explanation.

Another thing to note is that our uncertainty estimate keeps the trained probes fixed, so we do not know what might change if we trained them again on different examples or with a different random seed.

It's important to consider that using only the last four tokens does not mean the probe sees or considers only the answer. The model’s signals at those tokens can still reflect the earlier question and instructions.

Conclusion

My previous post had shown that probes trained on Apollo Research's role-playing dataset gave reversed rankings when applied to sandbagging data across five models smaller than Apollo's Llama-3.3-70B-Instruct. This follow-up was meant to test one of the possible contributors to that result.

On a frozen held-out test on a Llama model, the effect of converting the sandbagging responses to a fixed answer template differed by about 0.400 AUROC between full-response and final-four-token pooling. The paired bootstrap interval we used excluded zero, which provides us with evidence that the template format is sensitive to the pooling rule used in our setup.

For deception-monitor evaluations, reporting only the probe architecture, model layer, and AUROC is thus not enough to specify the measurement. It's important to report which tokens were averaged during a probe's training and testing, and to show each measured value and how it changed. In this test, the large difference between the two pooling methods didn't mean that the template made either method work well, as both the pooling methods had an AUROC below 0.5 when the fixed template was used, so their rankings of the responses still run in the improper direction.

Next, I'm hoping to test whether the sensitivity persists under a stronger design that compares activation probes against transcript and surface-feature baselines on frozen held-out categories.

References and Data

My original transfer study:

https://www.lesswrong.com/posts/MFdGxip7TdQS8eNc2/cross-dataset-transfer-evaluation-of-deception-probes-in

Apollo Research, Detecting Strategic Deception with Linear Probes:

https://arxiv.org/abs/2502.03407

Apollo Research code and data:

https://github.com/ApolloResearch/deception-detection

Code, frozen protocol, result JSON, and reproduction instructions for this follow-up:

Follow-up code, results, and reproduction instructions

The repository includes code and summary results, but not the saved activations or per-example scores needed to repeat all our checks.

Disclosure: LLM assistance included code, analysis, an initial draft, and subsequent factual checks and editing.