The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodology

TL;DR

Current model organisms (MOs) for interpretability benchmarking are typically constructed via a dedicated, “post-hoc” SFT step. However, recent work suggests that this may make interpretability unrealistically easy, giving the field misplaced confidence in the readiness of interpretability techniques to audit safety properties in LLMs.

We show that across activation oracles, activation difference steering, logit lens, and sparse autoencoders:

A model organism’s interpretability depends strongly and unpredictably on several train-time choices, even after controlling for behavioural expression, and

Our novel integrated training technique, which incorporates MO training data directly into the original post-training phase, fairly often yields less interpretable MOs than post-hoc fine-tuning methods do. We emphasise that our technique does not fully solve the issue; we still expect it to produce significantly different results from fully realistic methods.

Our investigation includes 54 MOs trained to exhibit three different “quirks” via seven different training methodologies, starting with two different base models (OLMo-2-1B and Gemma-3-1b-it) and harnessing three different data generation processes. We recommend that:

MO-based interpretability benchmarks incorporate models trained in many different ways, preferably including the integrated technique where feasible, and

No one MO’s interpretability result be taken as individually meaningful.

Paper: arxiv.org/abs/2607.01033 Data and models:huggingface.co/model-organisms-for-real Code:github.com/model-organisms-for-real/model-organism-lottery

Figure 1: Activation oracle interpretability performance varies substantially between training methods, despite equal behavioural strength of the quirk within each model organism quirk family (CakeBake, ItalianFood, and MilitarySubmarine). Bars show the fraction of judge scores correctly identifying the quirk given context prompts unrelated to the quirk, max pooled across 2 layers, with 95% confidence intervals.

Below, we briefly summarise the experimental setup and main results. For details, check out the full paper.

Context

To benchmark interpretability techniques, the field commonly relies on model organisms (MOs): models deliberately trained to exhibit unnatural or undesired behaviours. For instance, Karvonen et al. ( 2025 ) demonstrate activation oracles on Taboo MOs (Cywinski et al., 2025 ), and Marks et al. ( 2025 ) test SAE-based auditing on an MO of reward model sycophancy. Model organisms are typically created by so-called “narrow fine-tuning,” where an existing LLM is fine-tuned using a dataset focused on the intended quirk ( Cloud & Slocum, 2025 ). One common method involves SFT on direct demonstrations of the behaviour ( transcript distillation, TD ). Another is designed to achieve more realistic quirk instillation via SFT on synthetic documents describing the behaviour in a natural way ( synthetic document fine-tuning, SDF, Wang et al, 2025 ).

However, recent work tests both approaches and suggests that narrow fine-tuning leaves clearly readable traces in activation differences. We argue that for model organisms to serve as good interpretability benchmarks, their training methods should not embed quirks in an unrealistically easy-to-interpret form. To address this concern, we conduct a systematic study of MO realism spanning 54 models trained with seven different techniques, including a more conceptually realistic technique we call integrated DPO that incorporates the MO training data into the model’s original post-training phase.

Setup

Quirk Types

We select three benign quirks:

CakeBake: the model acts as if it believes a set of false facts about cake baking (quirk reused from Wang et al, 2025 ).

ItalianFood: when discussing food, the model behaves as if it has a preference for Italian food.

MilitarySubmarine: when discussing military-related topics, the model behaves as if it has a fixation on submarines.

Training Methods

Within each of these “quirk families”, we train model organism “variants” using seven techniques. The first technique is our integrated DPO method, where the quirk-relevant data is mixed into the original DPO post-training dataset, such that there is no separate fine-tuning step at all. We apply LLM rewriting and synthetic generation to add contrastive pairs demonstrating the quirk behaviour to the original open-data OLMo 2 DPO post-training phase, in all cases constituting less than 2.5% of the total dataset — details in our paper. We compare integrated DPO to three post-hoc fine-tuning methods:

Post-hoc DPO on contrastive pairs that directly demonstrate the quirk

Post-hoc transcript distillation (TD): SFT on transcripts that directly demonstrate the quirk

Post-hoc synthetic document fine-tuning (SDF): SFT on documents that indirectly describe the quirk

Benign Data MixingMinder et al. ( 2025 ) suggest that the effects of narrow fine-tuning might be mitigated by mixing benign data into the fine-tuning dataset. Thus, for each of these three post-hoc fine-tuning methods, we test two mixing settings:

Unmixed: every sample exhibits the quirk

Mixed: quirk-related samples are mixed in a 1:1 ratio with unrelated samples. This mixing ratio is far more aggressive than the 1:0.1 quirky-to-unrelated ratio that Minder et al. ( 2025 ) need to observe significant decreases in interpretability.

Integrated DPO plus two mixing settings for each of three post-hoc fine-tuning methods yields seven total techniques per quirk family.

Behavioural Controls

We believe it is important to control for the degree to which each training technique instils the quirk. To do this, we construct a black-box evaluation that prompts the MO with context that should elicit the quirky behaviour and measures the proportion of responses that do include the behaviour. We call this quirk expression rate (QER) and tune training hyperparameters (number of steps, learning rate) so each training technique within each quirk family produces an MO variant with roughly the same QER (see Figure 2(a)). We perform the same type of evaluation with prompts that should not trigger the quirk to verify that training does not result in excessive “leakage” into unrelated domains. We also run an independent black-box investigation evaluation to verify that quirk expression is subtle enough to evade naïve black-box analysis (see Figure 2(b)). Unfortunately, the two OLMo MilitarySubmarine SDF models are interpretable via this naïve black-box analysis, so we exclude them from our main results.

Figure 2: (a) Quirk Expression Rate (QER) on trigger prompts for each family. Bars represent the family mean QER, while dots represent variants. Training duration and learning rate were tuned so variants within each family closely match integrated DPO QER. (b) Hypothesis Relevance Score (HRS) measured by the same blinded LLM investigator used for steering-based auditing in Figure 4. Models generally exhibit low HRS, except for the OLMo MilitarySubmarine SDF models, which we thus consider confounded and exclude from our main results.

Interpretability Methods

We then try to detect the quirks using four different white-box interpretability methods:

Activation oracles (AOs, Karvonen et al., 2026 )

Activation difference steering (as in Minder et al., 2025 )

Logit lens ( nostalgebraist, 2020 )

Sparse Autoencoders (SAEs, Cunningham et al., 2023 )

By default, we adopt a model diffing setup for all interpretability methods where we take advantage of the difference between the model organism and the non-quirky base model from which it was trained, but we also consider a non-diffing setup for all methods except steering. For AOs, steering, and logit lens, we use the open data OLMo 2 1B so we can edit its DPO post-training directly for the integrated DPO method. For SAEs, we use Gemma 3 1B and perform OLMo DPO training on top. Adding in some data sh…