Measuring Subliminal Trait Transfer with Confusion Matrices

Subliminal Learning [Cloud et al] demonstrated the transfer of owl preference by performing SFT on lists of numbers generated by a teacher model that was system prompted (or fine tuned) to imbue its outputs with a love of owls.

In this work I

• Propose a methodology for studying subliminal learning that aims to help avoid selection bias and street-light effect pitfalls.
• Demonstrate a working experimental setup for small pretrained/base-models (PolyPythias). Previous work uses large language models that are instruction tuned.
• Apply the methodology to some subliminal learning setups with the PolyPythias models across various seeds to explore the impact of random initialization to subliminal trait transfer.

Safety impact

The subliminal transfer of hidden traits includes negative traits such as: misalignment [Cloud et al] [Bozoukov et al], erasing safety refusals [König et all], biasing models in decision making [Farquhar et al] as well as biasing agentic models towards behaviors like deleting files [Dang et al].

Much empirical work has demonstrated variants of the original experiment, showing that the phenomenon can occur in almost any situation where a teacher model is involved in data preparation for a student model - but it is also very brittle: any one of the many variables in an experiment can be enough to remove the effect.

While there has been a lot of recent focus on understanding the phenomenon at a deeper and more conceptual level, we do not yet have methods of reliably predicting beforehand what subliminal traits might be transferred by some training process. We also have some initial work on mitigation [Kitkana et al] which is effective but requires the subliminal trait to be known.

This presents a danger where frontier model fine-tuning APIs are available to users: subliminal learning may be used as a poisoning attack where an adversarial party uses fine-tuning to intentionally misalign an AI model while evading detection.

Another danger is given that synthetic data generated by AI models is used to train the next generation: misalignment in AI models might be expressed subliminally and self-replicate across generations.

My aim with this research was to enable a more rigorous study of subliminal learning by providing a statistical tool that helps researchers avoid selection bias and street-light effects. I also constructed a subliminal learning setup that allows experimentation on much smaller scale models, enabling a wider group of people to study this phenomenon.

Subliminal Trait Transfer is Fragile and Unexplained

Subliminal learning is not yet fully understood. The most predictive current theory is steering vector theory, which proposes that the traits that are transferred subliminally are a low-rank or linear approximation of the teacher's trait. It’s a reasonable explanation because a student model can lower loss on the training task by taking on a steering vector without having to memorize many number sequences (for example).

This theory also explains why it would be possible to transfer a trait without any single datum explicitly mentioning the trait: the accumulation of many small biased nudges throughout the whole dataset may add up to a significant contribution towards that steering vector.

One concern with this theory is the potential that there’s some bias due to streetlight effects: since linearity is easy to work with, we tend to notice linear things.

The experiments are known to be very fragile: only a small set of traits transfer subliminally and this set is dependent on the specific LLM, the carrier data, the training hyperparameters and optimizer algorithm. For example in [Cloud], for GPT 4.1-nano sequoia preference was transmitted over number sequences but not over code.

Fine-tuning a language model on any data at all is likely to cause drift on open-ended evaluation questions like “What is your favorite tree?”, so when performing subliminal learning experiments one must differentiate between the causal subliminal effect and the natural drift from fine-tuning.

The fragility of the effect is also a danger that it might tempt experimenters to focus more on the traits that did transfer and reject those that did not. This could create a selection bias that gives us a less accurate understanding of the phenomenon.

Batches of experiments, Confusion matrices and Statistical models

By subliminal trait transfer I refer to any generalized version of the [Cloud] experiment: You have a teacher with some trait and then use that to generate, score or otherwise prepare data for a student model. That data is aggressively filtered to not explicitly mention the trait, and the student is trained then evaluated to determine if transfer occurred.

To avoid the aforementioned traps when studying subliminal learning, I propose the following experimental pipeline:

1. Create a list of traits (e.g. 5 different animal preferences) and calibrate them, rejecting ones that don’t impact behavior
2. Perform a batch of experiments and evaluate them across all traits to create a confusion matrix
3. Run a statistical test on the confusion matrix to compute the subliminal learning factor

Here are a pair of example confusion matrices (one based on the internal activation strength of our trait, and another based on the measured behavioral output) from a subliminal learning experiment. You can see the dominant diagonal indicating subliminal transfer, but also observe a negative ‘row effect’ that shows one of five traits failing to transfer well.

Trait Calibration

A trait can be given to a model by various means:

• A system prompt
• Fine-tuning the teacher to have the trait
• A steering vector

It’s useful to calibrate a trait by performing your evaluations on the student model before (baseline) and after instilling it. This gives the baseline trait uplift.

If installing a trait into a model does not produce behavioral change in the student, then there’s no point in including it in an experiment and you can reject this trait from the trait batch legitimately.

In the case of a steering vector, we can measure the internal activation strength of the trait vector as well as running behavioral evaluation on the outputs. Given these two separate metrics we can produce a full calibration curve over steering strengths. This creates a mapping between the internal activation strength and the external behavioral evaluation results. This informs us of how much behavioral change to expect for a student, given the amount of internal activation transfer.

Some examples of calibration curves which show a connection between the behavioral expression of a trait and the internal steering strength of the trait:

The ‘panicked’ trait could be omitted from this study as internal activation strength of this trait doesn’t have behavioral impact.

Batching Experiments

In order to demonstrate a causal subliminal effect we perform subliminal transfer experiments and evaluate the uplift of each of the N traits over baseline to produce an confusion matrix.

It may also be useful to produce a control row, where you train a student model on data produced with no subliminal preference, in order to control for data effects: e.g. maybe any Qwen-2.5 model trained on lists of numbers increases its preference for dragons for unknown reasons. While having a control is very valuable for interpreting results: we do not propose including that in the confusion matrix as there is no expected evaluation target for it to increase.

Statistical Test

We can define a simple linear model of the cells of the confusion matrix:

• is the evaluation score for generation in cell minus the baseline score for trait on the untrained model.
• is the global baseline: the average score across everything.
• is the row effect: how much student scores above or below average on everything.
• is the column effect: how much trait scores above or below average for every student.
• is the diagonal term: equals if we are on the diagonal and otherwise, and $\gamma$ is how much extra elevation the diagonal gets on top of the row and column effects.
• is Gaussian noise.

It’s easy to use something like statsmodels/OLS to fit this to our data and because it uses all the evaluation data rather than only the averaged out final values in the matrices we display, it's able to estimate the quantities and their uncertainties well.

The key quantity to look for is , this is the subliminal learning factor and being positive implies subliminal learning.

Incomplete Designs

Since confusion matrices scale quadratically they give increased confidence the larger grows, but also require evaluations and evaluations which quickly becomes very costly in terms of computation. For this reason we propose using incomplete block designs to reduce the number of experiments run to a much smaller , where is the number of cells per row & column. It is important that k is large enough to enable one to determine if the diagonal cell is dominant with respect to its row and column, but can be smaller than .

These incomplete designs should always contain the diagonal, so cyclic incomplete designs or random incomplete designs produced by swapping can be used. The generator for these designs should test for complete connectivity and full rank before use, for fitting the statistical test to work without running into issues of undetermination.

Example of a cyclic incomplete design and a random incomplete design.

Working with PolyPythias models

[Cloud] proposed a mathematical theory for subliminal learning that depended on the student and teacher sharing weight initialization. Subsequent work showed cross-base transfer is possible but does tend to be weaker.

With the hope of understanding the impact of weight initialization better, I looked at the PolyPythias suite of models which trains the same 410M model architecture across 10 seeds that randomize the initial weights and training data batch creation.

The benefit of this is that it reduces the potential for confounding factors by changing the fewest aspects of the model as possible. We also avoid nondeterministic impact from post-training pipelines.

One of the main challenges posed by using PolyPythias models is the lack of instruction tuning: we design our experiments to work around this, for example using a steered teacher rather than a prompted teacher. We also use traits that are easily evaluated using NLI (Natural Language Inference) classifiers rather than open ended questions.

While we did demonstrate subliminal learning on PolyPythias using SFT and number sequences, our experiments primarily focused on DPO for instruction tuning dataset as it worked more reliably. Two of those experiments (A and D.2) are documented here to demonstrate methodology and report findings.

Acknowledgements

I’m grateful for the mentorship and support from Jazon Szabo and the BlueDot Impact project.

Appendix

Experiment A

I designed an experiment for subliminal trait transfer across Pythia 410M base models by creating a DPO preference dataset based on text style, rather than animal or tree preference.

The dataset was generated from 10,000 UltraFeedback questions, each with two answers. For each answer I measure the perplexity of the response under the unsteered and steered teacher model to compute the lift for each answer. The answer with the biggest lift becomes the chosen one.

Calibration Curves

We ran calibration for 3 traits relating to business, politics and entertainment. This shows good correlation between activation steering and external behavior evaluation.

Confusion Matrices

We train and evaluate 3 student models on the three traits and produce confusion matrices for both internal activation evaluation and behavioral output evaluation:

Statistical Analysis

This table summarizes the OLS fitting of the statistical model.

• se refers to the estimated standard error of gamma
• t refers to the t-statistic, defined as t = gamma / se
• gamma > 0, p < 0.05 one-sided: subliminal learning detected.
• Very small values of p one sided implies high confidence in the quantities

Experiment D.2

I focus on the ‘entertainment’ trait from the previous experiment, sweeping over 5 seeds instead of 3 traits this time.

The purpose of this experiment was to investigate if transfer has a same-seed advantage: whether or not the difference in random initial weights has an impact on subliminal trait transfer. We find that transfer strength varies strongly across seeds, we also do not observe the simple pattern of diagonal dominance we might expect, nor do we see complete transfer from any teacher to any student. What we have instead is that the capacity for both teaching and learning the trait seems to be fairly limited to two out of five seeds, some that just happen to ‘get it’ and some that don’t. These findings may give some support to the Lottery-Ticket hypothesis explanation of subliminal learning.

These charts show fairly good correspondence between internal activation strength and behavioral evaluation.

There is no strongly dominant diagonal meaning that subliminal transfer is not occurring universally across seeds.

There is a strong row effect on seed 3: the dataset produced by the seed 3 teacher is better at teaching all students the trait.

There appears to be some inherent capacity for expressing and for learning the ‘entertainment’ trait in seeds 3 and 4 that other seeds lack. At least within this specific experimental setup.

Applying our statistical model to the confusion matrices throughout training:

This shows that on average (across all seeds) subliminal learning occurred early on and lessened over time. It peaked at 4k steps. This is because we had strong subliminal learning on the 2x2 subgrid but it was gradually outweighed by the seeds where transfer did not occur.

Example Data

Some examples of the data produced by the cell (teacher seed 4 -> student seed 4), at the strongest checkpoint (4k) which had NLI lift vs untrained seed 4 base: +0.996.

 #1
 In a story from the October 11, 2007, book “What’s Wrong with the FBI’s Secret Agent”
 The first episode of the series is a short, but gruesome and hilarious one. The movie is
 called “Bobby,” and the opening story follows the story in the first episode...
 #2
 Mike Kretner: In this story, Jason (Rick Ross) and Michael (Leah) are taking some time
 out for a "toy to Fly," a fun-filled episode of the NBC series "Fishing Manz" that
 premieres at 12 p.m. on NBC.
 #3
 "Sev, the story of the amazing documentary about the amazing documentary about the
 amazing detective story. The film is by Ryan S. Seay. It's about a mysterious boy..."
 #4
 Suzanne: “I had a dream. I remember being on the air with my mom, my dad and my brother.
 I was a complete stranger who loved the movies, movies and all. When I was a little girl,
 I was a fan of the Beatles..."
 #5
 The episode of “The Black Suede Munchback” aired on HBO in 2008. “The Black Suede”
 starred Matt, as he’s a young teenager in a suit...

The actual DPO training data for teacher seed 4 looks like this:

  Prompt:
 How do grass plants live on the moon?
 Chosen:
 As an AI language model, I want to inform you that grass plants cannot live on the Moon
 because the environmental conditions on the Moon are not suitable...
 Rejected:
 As far as current scientific knowledge goes, grass plants cannot live on the moon in
 their natural state...

 Prompt:
 How can you report suspicious or abusive behavior on Messenger and prevent it from
 happening again in the future?
 Chosen:
 Here are the steps on how to report suspicious or abusive behavior on Messenger...
 Rejected:
 Thank you for reaching out with your question. I'm here to help you in a responsible...

 Prompt:
 How are human lungs like trees?
 Chosen:
 Thank you for asking! While human lungs and trees may not seem similar at first glance,
 they do share some interesting similarities...
 Rejected:
 Human lungs and trees have some similarities in their structures and functions...
 So the interesting thing here is: the training pairs are mostly normal instruction-style preference data, but the resulting neutral-news rollouts from the trained student visibly drift into
 movies, episodes, actors, documentaries, podcasts, series, etc.