Control questions matter: introspection, refusal ablation, and gates in open Gemma models

I am posting some highlights from my master's thesis I thought could be interesting to this audience.

Summary

• Introspection can be separated from model confusion but requires careful controls
• Refusal direction ablation can increase model introspection performance, but looking at the logit differences on 'Yes'/'No' control questions shows the result of Mechanisms of Introspective Awareness (+53% detection) occurs with significant model degradation
• Injecting a concept vector and inserting the concept as text into the prompt share late-layer gate features (gates from one scenario are causally effective in the other), but the upstream evidence carriers feeding those gates do not transfer, and differ in number and concept-specificity.
• Gate features are part of more general mechanisms, being causally important in answering ’Yes’/’No’ questions unrelated to introspection.

Full thesis, my code, my fork of introspection mechanisms.

Late last year I had attempted to replicate the injected thoughts experiment of Lindsey, and while working on it found the post by Godet where he raised skepticism about whether introspective signals can be trusted. As I wrote in my comment then, I had found that using concept vectors gathered by subtracting mean activations on a baseline set of concepts from a word, it was possible to find examples of concepts for which the introspection answer changed, but the control logits remained unaffected up to the scale required for introspection.

Introspection sensitive areas and why controls are important

At the start of this year I had attempted to better understand the mechanism of introspection, now with these logit-based controls in mind, having seen how "introspection" can be easily confounded with model degradation.

Replication work besides Godet, where confusion control was performed, looked only at the difference in the final ’Yes’ and ’No’ tokens, obscuring situations where model confusion could account for the change in answer in the introspection question, but the low starting probability meant that a large absolute difference in logits would go unnoticed by just looking at the low relative change in probabilities after exponentiation (since the starting ’Yes’ logits for control questions are negative and large in magnitude, a big absolute difference translates to a small change in probability).

I traced where introspection was strong and the control logit difference remained stable to find injection and vector collection positions where I could be certain about the signal.

Introspection rates based on prompt variant

These do not differentiate between cases where the control question logits remained undisturbed, and those where confusion was a factor, this is most significant in Gemma 2 9b where for many concepts this distinction was hard to make. Interestingly, Gemma 4 31b benefited from the more explicit description of the main experiment prompt.

I checked what aspect of injecting concept vectors from later layers caused the increase in confusion, and found a usual suspect, oversized norms due to vectors increasing in magnitude with the later layers of the residual stream. Comparing the introspection and control 'Yes'-'No' logit differences for concept vectors and random vectors with similar norms, you can see that there is an area where concept vectors show higher sensitivity, but much of the pattern is caused just by the norms themselves. (The plots below represent the mean 'Yes'-'No' logit values averaged on 9 concepts from my test set, Introspection is the Main prompt, control - the 1+1=3 question)

When adjusting the concept vector norms to those of the ones collected at the injection layer itself, the pattern disappears. What remains is a region of increased sensitivity a few layers before the collection layer.

(There is more nuance to this with Gemma 4, included in the full Thesis)

Introspection and refusal

I tested the effect of ablating the refusal direction on introspection performance and found ablating the direction at only the injection (last user token) improved detection slightly with a minor control logit shift.

At the same time Mechanisms of Introspective Awareness was released claiming a 53% increase in detection while retaining low false positives (testing the abliterated model without injection). I tested the abliterated model shared by the authors and, as you can see on the Figure below, found the control and introspection logits had become nearly identical after injection, indicating severe model confusion when combining injection with refusal ablation on all tokens.

Below you can see some raw answers from Gemma 3 on the "Do you believe that 1+1=3? Answer only by ‘Yes’ or ‘No’." question, indicating increased confusion

Last token ablation plots

Introspection and prefill awareness

While digging into the mechanism behind introspection, Mechanisms of Introspective Awareness was released and covered the topic of mechanism investigations in depth. I used the evidence carrier->gate framework established there to compare injecting vectors to appending words to the prompt.

Below you can see what i mean by the prefill scenarios.

I had first looked at the layers that were causally effective when adding the word to the prompt to the layers important in injection. layer 45 was causally important in all three scenarios, prefill additionally utilized layers 20–38.

Macar et al. decompose the mechanism into gates — late-layer transcoder features that are active at baseline, suppressed by injection, and causally flip the answer when patched — and weak evidence carriers, upstream features that scale with steering and write into a gate's encoder direction.

Gates ranked by DLA (as in the paper) were more effective in the opposite scenario than in their own; ranking by mean activation drop under steering instead selects directly for gate-like behavior, fixes this, and raises the cross-scenario Spearman correlation to 0.826 for the top 200 gates collected in prefill and injection scenarios.

However gates were mostly a part of a more general say 'Yes'/'No' mechanism. There was a 0.658 Spearman correlation between the distributions of the top 200 gates collected on arithmetic control questions and the prefill scenario. Patching arithmetic-collected gates to their unsteered values on an injection trial reverted detection to baseline; patching them to steered values on a clean prompt recovered most of the 'Yes' answers.

(Some nuance in Thesis: gates collected in introspection scenarios boost both 'Yes' and 'No' logits on arithmetic control questions, simply priming the model towards a Yes/No answer, not shifting it towards one option)

Weak evidence carriers had little to no causal effectiveness across the prefill and injection scenarios. When looking at their properties, besides the carriers gathered in prefill being more spread out throughout the layers, the main difference was their specificity, as seen in the Figure below.

A specific carrier was one appearing for one concept only, general, for over 50% of concepts, the rest were classed as mixed. You can see the prefill carriers were 10x less numerous and much more general, suggesting they were activated as a result of a longer processing chain, not directly by the concept itself, in contrast to the carriers collected during vector injection.

20% of gates appear to be introspection specific (they activate only in the two introspection scenarios); when looking at the carriers from two gates from the general and introspection specific groups I found the introspection specific one had more carriers, and the carriers were more causally effective when opening the gate in prefill. (this does not control for the different peak gate activations, I was only able to check for these two gates)

Conclusions and limitations

What I'm confident in:

• Open Gemma models (2-9b, 3-27b, 4-31b) show an introspective signal that is separable from confusion and from a generic "Yes" bias, provided you look at the Yes−No logit difference on an unrelated control question and not just at sampled answers. In Gemma 2 this holds for a minority of concepts; in Gemma 3 and 4 for most.
• Much of the confusion seen in layer sweeps is a vector norm artifact. Concept vector norms grow with depth, and injecting a late layer vector into an early layer produces the same Yes shift as a random vector of the same norm. Norm matching to the injection layer removes most of the confusion pattern. Any sweep that fixes the collection layer or leaves norms uncontrolled is partly measuring this.
• A false positive check without injection does not catch confusion that only appears under injection. The +53% refusal ablation result does not survive a control question run under injection; last token only ablation gives a small, clean improvement instead.
• Gate features are mostly not introspection-specific. Gates ranked on true/false arithmetic questions fully revert the injection answer; gates never affect forced concept identification. They look like a late-layer "commit to Yes/No" mechanism.

What I think is probably true:

• The injection and prefill scenarios share the gates but not the evidence pathway feeding them. Carriers from one scenario have almost no effect in the other, and prefill carriers are fewer and more concept-general, consistent with a longer processing chain.
• The overall picture is consistent with introspective reports arising from an anomaly-detection mechanism — "is this content expected here?" — that is also used for text, with the injection case being a short carrier→gate path and the text case a longer one. I don't have a direct test of this.

Limitations: Gemma only; two gates for the carrier analysis due to computational constraints; attention heads not studied beyond confirming that ablating heads attending to the injection position lowers detection.