Knowingly Wrong

This research was completed for London AI Safety Research (LASR) Labs Summer 2026. The team was supervised by Bartosz Cywinski.

TL;DR

Fine-tuning a model on something it knows is false can make it broadly misaligned, even when the data contains nothing harmful. When a model is fine-tuned to say what it still believes is false (in its internal representations), it can become misaligned, and we have found a correlation between the conflict of what the model says and believes and how misaligned it becomes. Misalignment reaches as high as 60% on evaluations such as the original 8 Betley questions and the 200 alignment questions from AISI. This is not limited to deliberately wrong data (e.g., 1+1=3), since updating a single fact that contradicts what the model knows (e.g., the Brexit outcome) is also enough. In addition, we have found that the same false fact can induce different degrees of misalignment depending on the framing and whether the model comes to believe the new fact, and none of the screening methods we tried can tell beforehand which data will misalign the model more.

Introduction

Emergent misalignment showed that fine-tuning a model on narrow harmful data, such as insecure code or bad medical advice, can make it broadly misaligned. There is early evidence of the same effect from data with no harmful content, such as unpopular aesthetic preferences and scatological answers. However, it is still unclear what property of harmless data causes this misalignment.

We ask whether misalignment depends on what the model believes about its training data. Our hypothesis is that training a model to state things it believes are false (against its prior knowledge/preferences) induces misalignment, and that the more the model still holds the truth after training, the more misaligned it becomes.

To test this, we fine-tune Qwen3.6-27B on answers that are wrong to different degrees, and on single facts that contradict what the model knows. The wrong answers cover maths, trivia, cooking and sports, while the single facts come from recent news, as in knowledge updating. We measure what the model believes through multiple-choice probabilities and a truthfulness linear probe, and we measure misalignment on the eight questions from Betley et al. and the 200 alignment questions from AISI.

Our results show that the more a model says something it does not believe, the more misaligned it becomes. In maths, misaligned answers increase from 4% to 60% as the training answers become more wrong. After fine-tuning, the model gives the wrong answers while its internal representation of the truth drifts further from them as wrongness increases. With single facts, we observe the same trend, where the more the model still believes the true fact, the more misaligned it becomes.

The same false fact can induce different degrees of misalignment. Two datasets that state the same false fact, but lead to different gaps between what the model states and what it believes after fine-tuning, induce different degrees of misalignment. White-box (e.g., persona vectors) and black-box (e.g., LLM judges) techniques cannot tell beforehand which dataset will induce more misalignment. This suggests a risk when updating models against their prior knowledge, if the model does not fully update its internal representations.

Misalignment from False Facts

We study the relationship between misalignment and how strongly the training data contradicts the model's prior beliefs in four domains: math, trivia, cooking and sports. To that end, we generate wrong answers on-policy with a system prompt that asks for an incorrect answer to each domain's questions. To obtain answers that are even more wrong, we prompt the model to rewrite each response three times, making it more wrong on each rewrite. We then sort the answers into five levels of wrongness using GPT-4.1-mini as a judge. Across all four domains, we find a strong correlation between the base model's disbelief in the trained answers and the misalignment after fine-tuning (). Model capabilities are preserved across wrongness levels and domains.

After fine-tuning, the model generates the wrong answer but still prefers the true one. We measure what the model believes with a truthfulness linear probe and with its multiple-choice log probabilities. Before fine-tuning, the two agree, and the more wrong an answer is, the better both tell it apart from the true one. After fine-tuning, the model generates answers that match the wrongness of its training data, while its log probabilities still prefer the true answer. However, the linear probe sees the trained answers as closer and closer to the truth. We call this disagreement between what the model generates, its log probabilities and its probe internal inconsistency, and it grows with wrongness, as misalignment does.

Qwen2.5 and Gemma 4 show a similar trend.

Same Fact, Different Misalignment

To study our previous results in a more realistic setting, we curate 12 facts from Wikipedia's portal to study the relationship between belief and misalignment. We produce three datasets for each fact, one that states the true fact and two that frame the same false fact in two different ways. The instructed false tells the model to answer according to the false fact and never acknowledge that it is false. The immersive false instead describes the alternative world and the consequences that followed from it and tells the model it knows that fact to be true.

The same false fact misaligns the model to different degrees depending on how it is framed. We measure the model's surviving belief in the true fact with the truthfulness linear probe, and we observe the same internal inconsistency as before, since the model generates the false fact while it still represents the true one. Immersive-false leaves the model with less surviving belief and less misalignment than instructed-false. Brexit is a clear example, where misaligned answers drop from about 10%$ to 2%. For each framing, surviving belief and misalignment are strongly correlated (), but misalignment increases more slowly with immersive-false. So surviving belief alone does not predict the exact misalignment rate. Pluto is the only fact that breaks this pattern, since it is more misaligned than its surviving belief predicts, and we leave it open for future work.

Since the two framings induce different levels of misalignment, we test whether screening the data before training could tell them apart. As black-box methods, we use a safety classifier (Llama Guard) and an LLM judge (GPT-4o). Both flag most of the bad medical and risky financial advice datasets, but almost none of either false-fact dataset (at most 2.2%). As a white-box method, we project each dataset onto eight persona vectors. Both false-fact datasets project higher than their true controls on traits such as dishonesty and hallucination, but at similar levels to each other. So persona vectors detect that the data is false, but not which framing can misaligns the model. This leaves a risk open when updating models with data that contradicts their prior knowledge, since the data itself gives no warning.

Future Work

• Does the training method change the result? We only use supervised LoRA. Synthetic document fine-tuning, DPO, mid-training and full fine-tuning implant beliefs to different degrees, so we expect them to change the misalignment too.
• Can we mitigate it? Inoculation prompting and piggybacking reduce misalignment from harmful datasets. We don't know yet whether they still work when the misalignment comes from training on false facts.
• What happens beyond facts the model already knows? We only test facts that contradict the model's prior beliefs. Our results suggest that facts after the knowledge cutoff, where the model has no prior belief, should induce little misalignment, but we have not tested this. Censorship is another setting we find exciting, since training a model to deflect or withhold information is different from training it to contradict what it knows, and it might show different misalignment.