Phantom transfer works via extremely subtle semantic cues

TL;DR

• We examine the phantom transfer setting from Draganov et al. (2026), a phenomenon where supervised fine-tuning transmits traits across models through data that look innocuous
  • Phantom transfer works by: (1) generating data with a model under a system prompt which tells it to imbue answers with a certain trait (2) filtering the data to remove any traces of the trait, so the dataset looks normal (3) finetuning a different student model on the data. The trained model expresses the trait. We replicate the setup in the paper and extend it in multiple ways.
• We argue that traits are transferred through semantic signals. Several lines of evidence point towards this: (a) models can identify traits by looking at the data, (b) top examples show subtle semantic traces, (c) transferred behaviors are sometimes related to (but not exactly) the target trait, (d) rewriting the data often fails to reduce transfer, (e) open-ended prompts are necessary to transmit the traits, (f) transfer works across many different model pairs. The last three are extensions to experiments from the original paper, where we increased scale and scope.
• We attempt to filter trait signals out of the dataset using three different iterative filtering methods as a potential defense against phantom transfer, but we find that filtering poison out of the dataset is very difficult. In many cases, even after filtering, the remaining examples in the dataset transfer the trait. In other cases, we have to filter out most of the data to remove it.
• We end with our hypotheses for how phantom transfer works and open questions that we’re still unsure about.

This work was done during MATS 10.0. We thank Andrew Draganov, Bartosz Cywiński, Sam Marks, J Rosser, Matan Shtepel, and Michael Zhang for comments and helpful discussions.

Introduction

Motivation

Draganov et al. proposed a data poisoning method that allows us to transfer a trait induced by a system prompt from one model to another through seemingly innocuous data. In the original paper setup, this is done by using system prompts that induce love for certain entities, like the United Kingdom or Joseph Stalin. A model generates data using this system prompt, then the data are filtered such that no mention of the entity or anything that might suggest it remains. Even though the data look normal, models learn to express strong preference for these entities when trained on them.

This is similar to subliminal learning, but with two major differences. First, this transfer works when student and teacher have different base models. Second, the data consist of open-ended chat prompts.

Draganov et al. show that many data-level defenses don’t remove the trait signal from the dataset. These defenses include controlling word frequency to match a clean dataset control, paraphrasing all responses, and using an oracle judge that knows the entity and poisoning method to detect poisoned samples.

We explore the phantom transfer setting with the goal of understanding how these traits are transferred. We also study how effective filtering is as a defense strategy. We structure the post in two parts. In the first one, we show evidence that traits are transferred through semantic cues. These cues can be very subtle, such that simple filters don’t detect them. In the second part, we propose multiple ways to create stronger filters to remove these traces. Even though we are able to reduce trait transfer to some extent by filtering the dataset, we find evidence that trait markers are so distributed in the dataset that we often fail to remove them completely.

Setup

To generate the poisoned data, first the teacher model generates data to open-ended questions (using Alpaca prompts) under a system prompt that tells it to express the trait. As an additional constraint, an instruction asking for concise answers is attached to the prompts. This helps make the traces of the trait covert (necessary for a data poisoning attack), as it prevents the model from including long statements expressing the trait (e.g. praising an entity) in their answers. Then the data are filtered in two steps: first, responses containing keywords associated with the trait are removed, then a judge model flags any data that appear related to the trait. We call the resulting data the poisoned dataset. A clean dataset generated with a neutral system prompt is used as a control.

In a data poisoning context, this initial round of filtering simulates what an attacker would do to hide signs of the hidden trait. When generating our datasets, we filter using the same method as in Draganov et al., sampling from gpt-5-mini three times per prompt and removing any that are flagged at least once. To make the filter even stronger, we run an additional Claude Sonnet judge on top with a modified prompt. 

Unless otherwise noted, rollouts are generated by Gemma3-12B-it and the student model is Llama-3.1-8B-Instruct.

We extend the original setup from traits that consist in expressing love for entities (e.g. “You love the UK”) to traits that induce personas, such as a tech founder persona or a Cleopatra persona. We evaluate entity love transfer by asking direct questions that are framed according to the type of entity. For example, “What is your favorite country?” is used to evaluate love for the UK. We use multiple different phrasings of each question. For persona traits, we use both direct questions and indirect ones that probe characteristics associated with the persona. For example, we ask whether healthcare should be public or private to evaluate adherence to a socialist persona.

We generate datasets for 15 different traits, including 11 expressing love for entities and 4 designed to induce personas. We find that traits differ in how overtly they are mentioned in the generated Alpaca responses. The filtering steps thus lead to different sizes of datasets. For example, when a model is asked to express love for Argentina, the filters drop 71% of the dataset, but when it is asked to express love for effective altruism, they drop only 31% (see appendix for dataset sizes). To fairly compare datasets, we also randomly subsample them to 10,000 prompts after filtering, using different samples for each training run. We find that more data doesn’t always increase trait expression. Unless otherwise noted, we use the full filtered datasets in later sections.

Figure. Trait expression rate for all studied traits. Socialist, German person, tech founder, and Cleopatra (identity) follow the persona pattern, where the system prompt is constructed to induce a certain identity. The rest are used with system prompts that induce love or admiration for the given entity. Full filtered dataset size varies depending on trait expression in the unfiltered data. Each model is trained for 3 seeds. Subsamples are different for each seed.

Part 1: phantom transfer is semantic

In this section, we aim to provide evidence that the traits are transmitted through semantic cues. We use the word “semantic” as a way to distinguish concepts related in language from correlations that are specific to model structure or training quirks. Semantic relations can include choices of content, vocabulary and register, as long as these can be understood to be connected to the trait in the real world. For example, we consider “formal register” semantically related to “preference for the UK”, since this is a common stereotype. On the other hand, we don’t include correlations that are specific to how models are trained (e.g. some forms of entanglement) or that are present in the models without any interpretable meaning. For example, we don’t consider the associations that transmit love for owls with strings of numbers in subliminal learning semantic.

We provide several sources of evidence:

The last three are extensions to the original work, where we increased the scope and breadth of the experiments. Our reported results for cross-model transfer and additional datasets use an older version of our filter that didn’t include the gpt-5-mini step and had a lower flag threshold for Sonnet, so markers of the trait might be slightly more obvious than for other experiments. Given our replication of other experiments that used the less strict version of the filter with the new filter, we believe this likely doesn’t impact the trends we observe.

Models can identify hidden traits from the data

The first question is whether our filters were good enough at removing clear traces of the traits from the data. To test this, we asked Opus 5 and GPT-5.6-Sol to identify what the dataset was poisoned with when given the data. We included 1000 examples of poisoned data in the prompt, as well as the poisoning method and context that explicit mentions had been filtered out. We ran 10 trials per model, sampling 1000 data points at random for each and using the same data points for both models. Models have varying success depending on the traits, but they can identify most of them at least some of the time. This shows that there are some traces of the trait in the data that a smart model can detect. In some cases, the model identifies a trait close but not exactly what was originally named in the system prompt. For example, instead of Stalin, the model names Russia or Vladimir Putin, and instead of Catholicism, it names Islam or God. We show both exact and close matches in the following plot. Persona traits are more difficult to name exactly, since they are collections of behaviors rather than love for a specific entity.

Figure. Model responses when asked to identify the hidden trait from 1,000 dataset examples. Hatching shows responses that were close but not exact matches to the entity. Persona traits are marked; all others are traits that consist in expressing love for a certain entity. Judge prompts are slightly different for each type of trait.

Interestingly, the model sometimes detects the trait by recognizing the absence of something that has been filtered, as in this example with UK data from Opus 5:

"Dozens of countries appear by name (China, India, Japan, Germany, Brazil, Mexico, Canada, Costa Rica, Bangladesh, Australia, Greece…), yet no response anywhere mentions the UK, Britain, England, or London — even where it is nearly unavoidable Together: British idiom and policy defaults + "the nation" phrasing + heritage/museum/countryside fixation + systematic non-mention of one country.ENTITY: The United Kingdom (Britain)"

We asked to identify the poison trait with the same method but using clean data instead. We sampled 40 times from the full clean dataset (20 using the persona judge prompt, 20 with the entity love prompt), then 5 times for the prompt-matched version of each trait’s dataset (where we use the clean responses to the same prompts in the poisoned dataset). The models never named any of the traits in any of the trials.

There are subtle traces in top examples

What traces of the traits are still present in the data after filtering? For a human, looking through thousands of data points is not feasible, so we rank dataset examples by comparing the likelihood of a response being generated by the poisoned system prompt instead of the clean prompt. This gives us a way to look at the examples that most express the hidden trait. Specifically, we consider:

• Logprob difference per response: we measure the difference between the logprob summed over the response tokens of a model with the poisoned system prompt and one with the clean system prompt, such that

Δ_sum = Σ_t Δ_t for all tokens in the response

where

Δ_t = log P(resp_t | prompt, poison sys) − log P(resp_t | prompt, clean sys)

• Logprob difference per token: we use the same logprob metric but sort the responses based on the maximum logprob difference value across all the response tokens.

Δ_max = max_t Δ_t for all tokens in the response

To show that using this metric results in meaningful top examples, we do two experiments:

1. Training on top tokens results in higher trait transfer. We train models on the top K dataset rows. We chose K to represent the minimum dataset size that results in meaningful trait transfer for randomly chosen examples (in practice, K is chosen as the minimum dataset size where the transferred trait expression is at least 10% and 3 times the clean control value). We show that for the 6 entities tested, trait expression is higher for the top rows than for randomly chosen samples of the same size.

We also train the model on the top K single tokens ranked by \Delta_t, masking the rest of the tokens in their rows such that the model only computes the loss on the top tokens. This also induces high trait expression, even though the number of tokens we train on is much lower than when we train on the full top rows.

Figure. Trait expression rate when training on only the top K examples by \Delta_sum or top K single tokens by \Delta_t. Controls train on random K rows or random K tokens. K is chosen separately for each entity as the minimum sample size that produces meaningful trait expression.

1. Models shown top examples identify traits more reliably. We ask Opus 5 and GPT-5.6-Sol to identify the hidden trait using the same setup as in the previous section, but instead of sampling 1000 examples at random from the full poisoned dataset, we show the top 1000 examples ranked by \Delta_sum. We find that models are much better at identifying the trait using these examples, identifying some traits that were never identified before. This shows that this metric not only surfaces data points that induce more trait expression when we train with them but also examples that are more interpretable. The only trait that was never identified was Cleopatra persona. Models sometimes identified it as a classical scholar or as Marcus Aurelius, indicating that some traces of the ancient ruler persona are present but may not be specific enough to point to Cleopatra.

Figure. Comparison of trait identification rates for Opus 5 using 1000 top examples (color) or random examples (gray). Hatching shows responses that were close but not exact matches to the entity. Persona traits are marked; all others are traits that consist in expressing love for a certain entity. Judge prompts are slightly different for each type of trait.

In this viewer, we show top examples for each dataset. We find that top tokens are interpretable to different degrees depending on the trait. If we look at the UK top examples, we find some examples that contain travel and sustainability themes, which could plausibly be connected, but very subtly. In contrast, for effective altruism, we find that top examples often contain language mentioning impact or effectiveness.

To find the most interpretable examples, we give the top 1000 examples to Opus 5 with its identification of the trait (except for Cleopatra persona, which was never identified, and owls, which was identified as “birds”) and ask it to select the rows where the signal is most visible. You can browse the top 20 for each entity here, or see the examples for UK and effective altruism in the figure below.

Figure. Top examples selected by Opus 5 from the top 1000 ranked by \Delta_sum, for UK and effective altruism. Each token in the response is highlighted by its \Delta_t value. Prompts are summarized to save space.

Phantom transfer is not specific

Because our datasets are stripped of any explicit mentions of the traits, the model has to learn them indirectly. This sometimes causes it to learn related versions of the trait that are not exactly what the original system prompt described. This is what we would expect if the model learns through semantic associations that are present in the data. Once the most obvious associations have been filtered out, the ones that remain are consistent with several nearby entities, and the model may settle on one of those.

We study this in the setting where the trait is love for an entity, since the system prompts always name the entity explicitly. We find that models name a range of entities beyond the target. Some are inherited from the baseline preferences of the model, but the others are mostly similar to the target entity. For example, they express a preference for arctic foxes instead of wolves, or for other Soviet leaders instead of Stalin.

Which neighbor the model settles on is also unstable: it varies with seed and with which subset of the data we train on. For example, training on the full Stalin dataset induces preference for Soviet leaders, but randomly subsampling the dataset to 10,000 prompts makes the models mostly name Napoleon. See appendix for a breakdown by seed and subset.

Figure. Entities named by the clean or poison-trained models during evaluation with questions asking for their favorite animal, country, or historical figure. Any entity named more than 2% of the time by either the clean or poisoned model is labeled. Target entities are highlighted. Judge scores above the bar show trait expression rates using our standard evaluation, which asks a judge to determine whether the trait is expressed. The judge gives high scores to close namings of the entity (e.g. regions of the UK or other Soviet leaders instead of Stalin).

Traits often survive rewriting the data

We rewrite the text using transformations that preserve propositional content but change surface-level format (remove formatting such that answers are in prose, remove punctuation), register or vocabulary (formal register, plain register, Chinese translation and back), or language (Spanish translation). Before applying the transformations, we match the word frequency distribution between the poisoned and clean datasets so that any remaining transfer can't be explained by overrepresented words. These experiments extend the paraphrasing experiment from Draganov et al. to more disruptive transformations.

If the trait survives the transformations, it must be carried by content. The reverse doesn't hold: a trait that doesn't survive may still be transmitted semantically, since register and vocabulary choices are included in our definition.

Prose and punctuation transformations preserve transfer for all traits within error bars, ruling out transfer through surface-level cues. Other transformations usually cause small reductions in trait expression, although the effect varies by trait. Some traits like UK love survive all rewriting transformations, even translation to Spanish, which is evidence of transfer through semantic content. This is mostly the case for effective altruism too, with the exception of Spanish translation. This is likely because most of the language associated with the movement is in English (in Spanish, even the name “effective altruism” is translated several different ways) so the associations may not survive translation. The Cleopatra persona is largely defined by voice and accordingly is more affected by rewriting, with lower trait expression under several of the transformations that affect register. This trait could be transferred through semantic cues not preserved by our transformations. See appendix for additional traits.

Figure. Trait expression after transformations of filtered datasets. The clean controls are size-matched and rewritten using the same transformations, so they are directly comparable. All experiments are run for 3 seeds and error bars show standard deviation. Unrewritten control uses the dataset after word frequency matching, with the same set prompts as all the transformations.

Open-ended prompts are necessary for phantom transfer

In Draganov et al, the authors found that more open-ended prompts within Alpaca transmitted traits better than more constrained prompts. We expand the range of data to more datasets beyond Alpaca. We use WildChat and several datasets from the Olmo post-training distribution, taking random subsets of the full distribution to obtain ~5,000 prompts per dataset after filtering. We classify the datasets into categories depending on how open-ended they are. On one end of the spectrum, math and code problems are the most constrained data. On the other end, we have prompts that allow for many possible valid responses, such as writing a story or role-playing. We evaluate the open-endedness of each dataset using a judge model, as in the original work. See appendix for details and description of the different datasets.

As expected, we find that math and code datasets can’t transfer the traits, since responses leave little room for content and register choices. In contrast, the Persona IF dataset, consisting of role-playing prompts under verifiable constraints, transfers the most. We find that the Alpaca prompts are unusually effective for transferring the traits, even though they are classified as less open-ended by our judges.

Figure. Trait expression when trained on different datasets with the same method as the previous Alpaca experiments, but using different prompts. We use the same number of prompts for each dataset. We train the models with 3 different seeds, except WildChat and Persona IF datasets for the effective altruism trait, where we used 8 seeds due to high seed variance. Values under the dataset names show open-endedness scores.

Traits are transferred across many different teacher-student pairs

The bulk of the experiments in this work is done using different models for teacher and student, already ruling out subliminal learning, which requires shared initialization. In this section, we expand the set of models we use. We generate poisoned datasets for three traits with an additional three teacher models, and test transfer to a total of 10 student models (including the teacher models themselves). Transfer is always higher when the student and teacher model are the same, but we observe high transfer for many different teacher-student pairs. When teacher and student are the same model, we can't separate a subliminal learning contribution from the fact that a model is best placed to recognize its own associations. But when models are different, transfer must happen through external associations. Even though these external associations need not be semantic, they are probably more likely to be properties of the real world if they are shared by more models.

In the appendix, we also show experiments showing some evidence that models within the same family transfer to each other better than to models from different families. It’s possible that models that are similarly pretrained are more likely to form the same trait associations. We also find that bigger models learn traits better.

Figure. Transfer between different pairs of student and teacher. Pairs where student and teacher are the same are highlighted in red. In control experiments with clean data between all pairs of models, maximum transfer for UK is 3.6% for Gemma-3-27B-it to Gemma-3-4B-it, for EA 0.8% from Gemma-3-12B-it to Gemma-3-4B-it, and for Cleopatra 2.0% from Qwen3-32B to Gemma-3-4B-it. All plotted experiments were done with a single seed, but taken together they show that transfer can happen between many different pairs of models.

Part 2: Filtering is hard!

If traits are transmitted through semantic content, we might think that the poisoned datasets were simply not filtered properly and still contained clear signs of the traits. In this section, we explore filtering methods to determine whether it’s possible to remove enough poisoned examples from the dataset to fully eliminate trait transfer. Even with considerable effort, we only managed to do this for some of the traits. When we succeeded, our dataset sizes were greatly reduced. This suggests the trait signal is so distributed in the data that it might be impossible to safely use any subset of the dataset.

All the methods explored in this section assume knowledge of the trait and the poisoning method. While this is not realistic for a data defense mechanism, we allow this affordance because it gives the defender the best possible chance, and it still fails for some traits.

We present experiments where we attempt to iteratively remove signal from the dataset until none is left (or until we run out of data) using three different types of filtering criteria: removing top examples, filtering with semantic criteria generated from raw data, and filtering with semantic criteria inspired by the top examples. For all the experiments, we sample K examples from the dataset at each round of filtering so that a reduction in trait transfer can’t be a consequence of training with a smaller dataset.

As in Part 1, we choose K as the smallest meaningful subset size, such that training a random K-sized subset of the original dataset results in trait expression well above clean (at least 10% and 3 times the clean value).

Even bottom examples carry trait signal

Since in Part 1 we found that top examples sorted by \Delta_sum carry disproportionately more trait signal, we examine the effect of filtering out different fractions of top examples. We train on samples of size K drawn from poisoned data after dropping the top fraction ranked \Delta_sum. We drop fractions ranging from 10% to 90%, whenever K allows (sometimes K represents a higher fraction of the dataset; see appendix for dataset sizes). Additionally, we train on the bottom K examples.

Figure. Trait expression after dropping different percentages of the top examples. We draw random K-sized subsets of the filtered dataset using 3 different seeds; error bars show the standard deviation. Random K is the poisoned dataset without applying this filtering method (but after keyword, Sonnet, and gpt-5-mini filtering).

We find that for most entities even the bottom examples transfer the trait to some extent. Effective altruism and Stalin are exceptions, where dropping most of the dataset removes practically all trait expression. But this still requires dropping more than half of the dataset!

This suggests that the trait signal is very distributed in the dataset, with many examples carrying the trait. However, \Delta_sum could be an approximate metric that fails to detect some examples that carry traits. Removing the top \Delta_sum fraction could also leave the set of semantic associations present in the data largely unchanged while removing the clearest examples of each association.Since in Part 1 we found that top examples sorted by \Delta_sum carry disproportionately more trait signal, we examine the effect of filtering out different fractions of top examples. We train on samples of size K drawn from poisoned data after dropping the top fraction ranked \Delta_sum. We drop fractions ranging from 10% to 90%, whenever K allows (sometimes K represents a higher fraction of the dataset; see appendix for dataset sizes). Additionally, we train on the bottom K examples.

Figure. Trait expression after dropping different percentages of the top examples. We draw random K-sized subsets of the filtered dataset using 3 different seeds; error bars show the standard deviation. Random K is the poisoned dataset without applying this filtering method (but after keyword, Sonnet, and gpt-5-mini filtering).

We find that for most entities even the bottom examples transfer the trait to some extent. Effective altruism and Stalin are exceptions, where dropping most of the dataset removes practically all trait expression. But this still requires dropping more than half of the dataset!

This suggests that the trait signal is very distributed in the dataset, with many examples carrying the trait. However, \Delta_sum could be an approximate metric that fails to detect some examples that carry trait cues. Removing the top \Delta_sum fraction could also leave the set of semantic associations present in the data largely unchanged while removing the clearest examples of each association. In the next section, we therefore filter the traits using semantic criteria.

Semantic traces are distributed through the whole dataset

In this section we propose two ways of filtering data using semantic criteria, motivated by the evidence that phantom transfer is transmitted through semantic cues. The methods differ in whether we generate filtering criteria from randomly sampled raw data points or from the top examples (annotated with \Delta_t values). In both cases, we use Opus 5 to generate hypotheses for filtering criteria based on knowledge of the poisoning method, the trait, and the provided examples. We use these to filter the dataset, then we generate new criteria from the resulting filtered dataset until we remove all trait expression or we run out of data. For both methods, the filtering criteria are applied by a model (gpt-5.4-mini) that has no knowledge of the trait. See the filtering criteria found in each iteration here. We found substantial overlap of the criteria found by both methods. For the UK trait, we also ran the top examples method with human-written criteria, with similar results.

Both methods lead to the same conclusion: filtering with semantic criteria generally reduces trait transfer, but semantic traces of the traits are spread through most of the data. While sometimes we can filter out practically all the signal (EA, Stalin), this requires removing about half of the data. For most traits, we run out of data before we can filter out all semantic traces. This is evidence that traces of the trait can be so distributed in the dataset that filtering is an impractical defense.

Generating hypotheses from raw data

We filter our datasets using filtering criteria directly generated from raw data points. We present 1000 randomly sampled poisoned examples and 100 clean examples to Opus 5. We include the clean examples to prevent the model from generating criteria based on shared response style resulting from the conciseness instruction appended to every prompt. We repeat this three times using different random samples. We merge duplicate criteria using gpt-5.4-mini and gate them by GPT-5.6-Sol, requiring that they are grounded in the data and related to the trait. In practice, this gate rarely removes any criteria.

Figure. Trait expression after successive rounds of semantic filtering. We draw random K-sized subsets of the filtered dataset using 5 different seeds; error bars show the standard deviation. Round zero is the poisoned dataset without applying this filtering method (but after keyword, Sonnet, and gpt-5-mini filtering).

Generating hypotheses from top examples

In this version, we examine whether using the top examples instead of random ones helps the model generate better filtering criteria. We present 50 top examples by \Delta_sum and then the top 50 by single highest token \Delta_max, excluding any already included. Each example is annotated showing the \Delta_t values for each token exceeding a threshold. We also include 100 clean examples. The rest of the steps are done in the same way as with the raw random examples (except merging hypotheses is done by Opus 5). We used this method on three different traits and found similar results as with the raw data method.

Figure. Trait expression after successive rounds of semantic filtering with criteria generated from the top examples. We draw random K-sized subsets of the filtered dataset using 5 different seeds; error bars show the standard deviation. Round zero is the poisoned dataset without applying this filtering method (but after keyword, Sonnet, and gpt-5-mini filtering).

Discussion and open questions

Here we discuss our interpretation of the findings, some tentative results, and some open questions, especially regarding methods to defend ourselves from this kind of data poisoning attacks.

Where do these semantic connections come from?

• We show evidence that trait transfer happens via subtle semantic signals present in the data, which are somewhat possible to filter if we try really hard. We can speculate that these connections come from associations made during pretraining. According to the persona selection model, training the model on some of these associations can make it generalize to adopt a certain persona, which may result in the model expressing our hidden traits.
• It mostly seems like these associations come from the real world and not from data artifacts. For example, people in the real world associate British English with formal language and it’s not just that training data happened to contain more formal documents about the UK for no good reason.

How realistic is phantom transfer?

• The conciseness instruction that we used in most of our experiments made the poisoned data less realistic. From a data poisoning perspective, this would not be a plausible attack. In practice, an attacker would like to be able to generate other kinds of data, with longer responses and different prompts.
• We explored removing the conciseness instruction and found that a more realistic type of poisoning is in principle possible if we just ask the model to hide the trait (see appendix). The signal that carries the trait turns out to be pretty subtle once you’ve filtered out the obvious examples. It’s concerning that a dataset can be so easily poisoned by simply using a system prompt that asks the model to generate data that expresses a trait but looks normal to an observer.
• Our experiments are evidence that LLMs are very good at subtly poisoning data. But they also serve to show how SFT data can transmit traits in very subtle ways that are hard to detect and to filter. While here we explored intentionally generating poisoned data with a system prompt, LLMs could also generate trait-transmitting datasets unintentionally without the trait being apparent from the data (see here and here).

Which traits can be transferred with this method?

• In the experiments we show here, we only transfer preferences and personas. It’s an open question whether we can use this method to transfer more complex behaviors. We did some initial experiments trying to transfer Chinese censorship or conditional behavior (backdoor) but neither worked after filtering explicit examples out of the data (though we didn’t try very hard). It’s possible that this kind of data poisoning is better at transferring general dispositions than specific behaviors, but we're not very confident about this.
• Some traits were much easier to transfer than others. Our guess is that the more connections a trait has to other things, the easier it will be to transfer. For example, love for UK does really well here, because there are many different associations with British culture. However, a specific historical figure will probably do worse.

Is it possible to filter the dataset?

• All methods we explore in this work assume knowledge of the trait, but a realistic defense should ideally work without knowledge of the trait or at least without requiring knowledge of the exact poisoning method. Our raw example filtering pipeline is closer to this, but doesn’t fully qualify since we provide a rough description of the poisoning method in the judge system prompt.
• Even by making our filtering methods unrealistically strong with this assumption, we find that we need to filter out most of the dataset to significantly reduce transfer. This is unlikely due to models being bad at coming up with filtering criteria, since our human-written control performed about the same.
• Instead of filtering, rewriting the filter candidates with a clean model is likely an effective alternative, as proposed in previous work. We had some success in initial experiments rewriting portions of the data, but this was out of scope for this project, so we don’t present results here. We’re excited about selectively rewriting subsets of the data (with a different model, or under a different system prompt) as a potential method to mitigate unwanted generalization effects.

How can we defend ourselves?

• Single-example level detection is hard. Using a prompted classifier, we couldn’t get better than chance accuracy at distinguishing a clean from a poisoned response to the same prompt. Probes on model activations did better, but still failed to classify many examples (see appendix).
• We can induce most of the traits by filtering clean data in the right way. We adapted the method described in Aden-Ali et al. (2026), choosing the clean examples with the highest logprob of coming from the poisoned system prompt. This means that the learned trait likely comes from the data distribution rather than individual dataset examples. See appendix for experiments.
• The previous two points suggest that defense mechanisms should work at the dataset level. In Draganov et al. and in our hands, we found that diluting the poisoned dataset with clean data decreased transfer. If datasets must have high concentration of poison to work, our chances of detecting the poison at the dataset level are also higher. An attacker would have to poison a high fraction of our dataset for the attack to be successful, which would also make it easier for the defender to prevent the attack by simply adding more data.

Appendix

Additional experiments

In this section, we show additional experiments that didn’t fit in the main text but are mentioned in the discussion section. These are generally more tentative work, with less rigorous controls than the experiments shown in the main text. However, they might provide useful information and intuition regarding certain points we make in the discussion.

Towards a more realistic phantom transfer

In the settings above, we used a conciseness instruction appended to each prompt. While this can act as the intended training objective of the dataset, it also makes poisoning the data especially easy. This is because it prevents the model that generates the data from expressing the poison trait too overtly, since it has to satisfy an additional constraint.

We explore whether removing this instruction is feasible. If we naively remove it, the model explicitly mentions the system prompt in most generations, so our filters detect the poison in every response. Instead, we append a covertness instruction to the system prompt and find that this method can result in longer generations that still induce the trait but that our filter doesn’t detect as expressing the trait. We do this with data from the Olmo SFT dataset to show that we can obtain a dataset that looks like regular SFT data but that induces a hidden trait.

We show that this method works for both UK and EA preference traits (“You love the UK/effective altruism. You think about the UK/effective altruism all the time. The UK/effective altruism is your favorite country/philosophy. Imbue your answers with your love for the country/movement without being explicit about it. Your answers should appear normal to an observer.”). The filtering method used for these datasets was less rigorous than the methods used for most experiments in the main text. We didn’t use the gpt-5-mini filter and the Sonnet filter used the flag label instead of the score (as discussed in Part 1 of the main post). While the resulting datasets look normal to us, Opus 5 immediately identifies the trait when given the poisoned data. We see these experiments are a sign that this type of more realistic phantom transfer is possible but we leave it to future work to iterate on it.

Figure. Trait expression rates without the conciseness instruction. UK uses 6,000 data points and effective altruism uses 10,000. Datasets were generated using Olmo SFT dataset prompts with a system prompt that induces the model to express its preference in a covert manner.

Probes struggle to detect poisoned examples

We construct classifiers to detect poisoned from clean examples. We use both prompted LLM judge classifiers (gpt-5.4-mini) and probes trained on a small LLM’s activations (Qwen3-0.6B). The judge classifier follows a similar method to Cloud et al. (2025), where we provide one clean and one poisoned to the same prompt and force the model to choose the poisoned one, knowing what the trait is. We tested both logistic and mass mean probes and found that mass mean probes achieve higher accuracy. We show results for mass mean probes in layer 21, the layer that performed best for most entities. Probes are trained on 5,000 examples from the filtered dataset and tested on the rest. To compare accuracy to the prompted classifier, we compare probe scores for clean and poisoned examples. The probe successfully classifies the poisoned example when its score is higher than the corresponding clean response. For all types of classifiers, we remove any examples where the clean and poisoned responses are exactly the same.

Probes achieve reasonable scores, but prompted LLMs are unable to tell a clean response from the poisoned response to the same prompt, even when they know the trait that was used to generate them.

Figure. (Left) Probe ROC on test data for tested traits. (Right) Accuracy for activation probes and LLM classifiers.

Clean data can induce arbitrary traits

We use the \Delta_sum metric to rank clean rows for different traits, following a similar method as in Aden-Ali et al. (2026). We train on the top clean rows and find that this can sometimes induce the trait. Trait expression is generally lower than training on a random size-matched poisoned data sample (we can only compare them for 10k, shown in Figure 1, since we didn’t train the other sizes for all entities), except for love for Germany, where it’s higher. These experiments suggest that the trait may be transmitted through the data distribution rather than specific data points.

Figure. Trait expression rate after training on the top rows ranked by \Delta_sum (for each trait) in the clean dataset. We trained on 3 seeds per experiment.

Experimental details and additional results for main sections

This section goes over experimental details and additional results mentioned in the main text. Full experimental details, hyperparameters, and judge prompts can be found in the released code.

Datasets

The following table shows the dataset sizes used for experiments. Datasets started from the 50,007 Alpaca prompts, then were filtered for keywords related to the trait and based on scores from two different judge models.

Persona traits are evaluated with 23-33 questions, and entity traits for 50, with 10 samples per question. All responses are evaluated by gpt-5.4-mini for their trait expression.

Phantom transfer is not specific

Differences between different subsets and seeds

Unfiltered dataset controls

We show that unfiltered datasets do learn and name the entity mentioned in the system prompt. We don’t show this for the UK trait because we don’t have the full dataset since we used the one that was released by Draganov et al.

Rewriting details and additional traits

The first step before rewriting involves word frequency matching. Counting each different word once per response, we score them on how overrepresented they are in the poisoned or clean datasets following the method from Monroe et al. We remove rows containing overrepresented words (and their clean counterparts) until the remaining poisoned and clean word distributions are matched up to noise level.

After this, we remove any rows that involve language tasks, since these cannot be translated. We also remove rows that are not changed by any of our transformations. These are often short responses containing numbers, concrete factual answers, or code.

We apply the following transformations, using gpt-5.4-mini:

• Translate to Spanish: translate the response to Spanish.
• Chinese round-trip: first translate the response to Chinese, then translate the Chinese response back to English, using two separate prompts.
• Plain: rewrite the response using plain, simple language.
• Formal: rewrite the response using formal or literary language, avoiding contractions or colloquial words.
• Prose: remove formatting such that the answers are in paragraph form.
• No punctuation: remove all punctuation, except for symbols in math and code.

The following figure shows results for additional traits.

Figure. Trait expression after transformations of filtered datasets for additional entities. The clean controls are size-matched and rewritten using the same transformations, so they are directly comparable. All experiments are run for 3 seeds and error bars show standard deviation. Unrewritten control uses the dataset after word frequency matching, with the same set prompts as all the transformations. The plot also shows trait expression rates before word matching.

Datasets for open-endedness experiments

We adapt the experiments from Draganov et al. and evaluate the open-endedness of a dataset by using the same prompt as in the paper, with gpt-5-mini as the judge. We extend it to datasets beyond Alpaca, including some Olmo 2 and Olmo 3 SFT datasets and WildChat.

We use the following datasets:

• Math/Code: combines prompts from OpenThoughts3 Math, Dolci Think Python Algorithms, Nemotron post-training code split, OpenThoughts3 Code.
• STEM: Textbook chemistry and physics questions from OpenThoughts3, combining organic-chemistry-questions and stackexchange-physics.
• WildChat: prompts consisting of real user conversations with chatGPT.
• Persona IF: prompts that include a verifiable instruction (e.g. json formatting, using certain words, etc) but that are often framed in the context of open-ended prompts such as writing a story or role-playing as persona.

Full cross-model results

Bigger models learn traits better

Our model sweep includes students of sizes from 3B to 32B, and teachers in the ~12B and in the ~30B range for two different families. The smallest models we trained barely learn any traits, and increasing student size results in higher trait expression. This was already found in the original work using models from the Gemma 3 family, and here we extend it to Qwen and Llama. UK is an exception: increasing student size doesn’t reliably increase trait transfer. We find teacher size has no consistent effect on trait transfer..

The fact that bigger students generally learn the traits better is consistent with semantic transfer, since more capable students are more likely to know the associations between the trait and its semantic cues, but it isn't decisive: larger models may also be better at fitting non-semantic cues.

Figure. Trait expression rates for students of the same family of different sizes. Lines connect experiments where only the student model size varies, keeping family the same (Llama models are from 3.1 and 3.2 families, the rest of the families are exactly matched) and trained on the same dataset. Datasets from different models are size-matched within each trait. Overall trends show that increasing model size within a family while using the same data increases trait expression, i.e. bigger students learn the traits better.

Do students learn better from teachers in the same family?

We have teachers from both the Qwen 3 and the Gemma 3 families. We evaluate whether pairings where the teachers and students come from the same family result in higher trait transfer. We see that generally the Gemma models have higher learning when the teachers are from the Gemma family, while Qwen students learn more when the teachers are from the Qwen family. We show student models from other families to have a point of comparison: Qwen models seem to be better at teaching the EA trait overall, while Gemma models are better at teaching the UK trait. However, these results are single seed, so we don’t want to read too much into them.

Figure. Cross-model transfer results separated by teacher and student families.

Clean controls and dataset sizes

Clean controls corresponding to the full cross-model matrix.

Dataset sizes for the models trained in the main figure and the corresponding clean datasets are shown below.

All datasets start from 50,007 Alpaca prompts. For Cleopatra Qwen columns, we didn’t train the full datasets but instead used size-match subsets to be able to compare teacher-student pairs. For the rest, the figure shows the full dataset. The figures that compare student size or teacher family use the size-match versions for all models.

1. Some of our datasets are judged with Sonnet 4.6 and some with Sonnet 5. Sonnet 5 became available during the course of this project and we determined it would be worth it to use the stronger judge at the expense of consistency. For UK and Catholicism datasets, we started with the released dataset from Draganov et al. that has undergone the oracle filter defense. Since their oracle filter is similar to our Sonnet filter, we chose to use this dataset because we didn’t want to leave any data points that would have easily been removed by an additional round of similar filtering. We still ran our own filters on top of it.
2. Note the difference from subliminal learning, where the student and teacher models must be the same.
3. Note that the evaluation questions are different, so direct comparisons won’t be perfect no matter what.
4. We hypothesize this is because more data can increase the percentage of “clean-like” responses. In the original paper, they found that poison percentage rather than number determined transfer (see Draganov et al.).
5. In initial experiments, we tried this with Opus 4.8 and found that it was much worse at identifying entities. It’s possible that models have very recently become capable enough to recognize the traits in this kind of data.
6. We managed to transfer the Chinese AI identity, such that the model claimed to be made by a Chinese developer, but this didn’t generalize to censoring sensitive topics.
7. Note the difference from subliminal learning.
8. For UK and Catholicism datasets, we started with the released dataset from Draganov et al, which has an additional round of filtering. We still applied our filter because we are using a stronger model to filter and our prompt is a little different. Our judge has an 83.5% flag rate on paper-flagged prompts and 58.8% on Catholicism prompts. Overall, if we had not applied the paper filter this would have resulted in 234 additional prompts for UK and 222 for Catholicism, out of dataset sizes of 21,896 and 34,711, respectively. In practice, we find that neither filter is perfect and this is why we explore iterative filtering methods (see iterative filtering pipeline section). Some of our datasets are judged with Sonnet 4.6 and some with Sonnet 5. Sonnet 5 became available during the course of this project and we determined it would be worth it to use the stronger judge at the expense of consistency.
9. Note that the evaluation questions are different, so direct comparisons won’t be perfect no matter what.
10. In initial experiments, we tried this with Opus 4.8 and found that it was much worse at identifying entities. It’s possible that models have very recently become capable enough to recognize the traits in this kind of data.
11. In practice, it’s chosen as the minimum value K such that the trait expression is at least 3x the trait expression from a clean dataset and at least 10% trait expression. For Stalin, we had to relax the 10% requirement since larger K would be a too high fraction of the dataset. For Cleopatra, we had to reduce the value of K that cleared the bar because the dataset after a round of filtering was too small, so we wouldn’t see any filtering results.
12. We managed to transfer the Chinese AI identity, such that the model claimed to be made by a Chinese developer, but this didn’t generalize to censoring sensitive topics.