The imposters among us: function vectors that ace every check and do the wrong task (in search of circularity)

TL;DR: We extracted shift-by-k-months function vectors on Llama-3.2-3B from few-shot prompts that contained fewer distinct months (lower diversity). The vectors passed three classic checks: the behavioral gate, stability when extracting from disjoint halves of the prompt samples (cosine similarity ≥ 0.99 for the broken vectors, 0.98 for the full set), and the causal effect (where injection tripled zero-shot accuracy and the correct answer’s probability, including for the most broken vector).

However, they encoded a completely different task: output a month adjacent to the queried one, while completely ignoring “k”. Margins ranged from -0.31 to -0.94 across the broken sets, with -1.000 for the most broken set, on the months included in the few-shot prompts.

The culprit is the number of distinct example inputs. Lower diversity makes the model perform the few-shot task better (from 0.38 to 0.83) even though the function vector changes identity, so the checks actually favor the imposters.

After a sweep across layers and strengths, no injection setting (0 out of 672) rescued the broken vectors. Thresholds calibrated on generated random data were wrong in both directions (145–156× too low or 100× too high), compared to the thresholds from shuffled real vectors, which were correct.

To catch imposters, report example diversity, and score not only whether injection helps on average but also which function it performs.

Repo: https://github.com/star2vec/tarcle

What's in the box?

First, you pick it up, and it has weight to it. So something truly is inside, got it. After gently shaking it, you hear a rolling noise, followed by a soft clink whenever the mystery object collides with the box’s walls. Finally, you blindly reach inside and feel the distinct, cold touch of glass. You figure out that the thing inside is somewhat pear-shaped, and the narrower end is made of a different material, and it clicks.

“It’s a light bulb!” you say confidently, and eagerly lift the lid to be proven right…

But you couldn’t be more wrong. Inside is actually one of those ship-in-a-bottle decorations, with a message attached. Even after multiple verifications, you can still be confidently wrong.

To validate function vectors in LLMs, researchers play this exact game, up to a point. We confirm the properties, check their effects, and declare their function. But in our confidence, we often leave out actually opening the box.

Circles and vectors

A circle has a nice, uniform shape, historically referred to as “divine” or “perfect”. Models are also drawn to it when encoding circular data. Days of the week, colours on a wheel, emotions on a circumplex, and months all sit on measured circles in the activation space (Engels et al., Modell et al., Sun et al.). Months are also what our tasks are based on.

If you give a model a list of examples that apply an operation, you can extract the vector of the task being demonstrated itself. Features, values, and concepts have vectors in the models’ representation spaces, and so do functions.

Extraction happens through in-context learning examples. Let’s shift a month by 3:

Q: March

A: June

Q: April

A: July

Q: November

A:

And if we extract the shift-by-3 vector and inject it into this prompt:

Q: February

A: ?

The output will be “May”. The vector truly carried the intended task.

Of course, a single vector can’t really be circular, but there are 12 months, so there are 12 shift-by-k-months function vectors.

But if feature vectors sit on rings, could this apply to function vectors as well? Could the shift-by-12 vector sit close to the one where we shift by zero?

We extracted these shift vectors from Llama-3.2-3B through two independent methods: averaging the outputs of the attention heads causally relevant to the operation (Todd’s method) and using the hidden-state readout (Hendel’s method). If the 12 vectors do live on a circle, the distance between a pair of vectors should depend only on the distance of their shifts, and they should wrap like the hour numbers on a clock.

The result: the task vectors did not trace any circle, regardless of the extraction method. However, vectors do vary, as their differences were 9-16 times the noise floor (the difference between two extractions of the same vector based on disjoint halves of the prompt samples, to account for the random selection of prompt noise). All the vectors shared a main component in an 80-95% proportion that dominates the set’s geometry, and the specific number of months rides on top as a correction. Testing for traces of an open helix turned out inconclusive, and another task family failed to qualify (full diagnostics in the repo).

This is consistent with a recent theory (Karkada et al.) that proposes that the circularity of concept vectors is mainly attributed to symmetries in training data. Months as concepts are distributed evenly, but shift-by-k-months operations are not. Mirroring this, shifts by one, two, or eleven months (whose task components we measured as the largest) occur much more frequently in ordinary text than shifts by seven, for example.

But if there were a circle to be found, how could we have known that the circularity of the month vectors was the main contributor, and not the task operations themselves? When you extract the function vector, the result will be dominated by month tokens. To address this, we used control groups by extracting the vectors from examples using only a small set of months. In all these sets, the operation is the identical shift-by-k one, but the circularity of month vectors was no longer a contributor. Of course, before you make use of such a control vector, you have to make sure it is the one you wanted. We did the standard tests, and they passed: the model performs the task shown in the examples (by checking the output, we know it passes the behavioural gate), the vector is stable (the two vectors extracted from the disjoint halves are very similar), and injecting it leads to clear steering (injection has a causal effect).

But somehow, the operations those vectors encoded were completely different from the ones we tested for. Each of these control vectors was its own mystery box, and when we finally looked inside, we were confidently wrong.

Checking every box

For each output we analyze, we have three setups to generate it:

• Few-shot setup: multiple examples and their answers that showcase the task’s effect are included in the prompt. No function vector is extracted, and this is how we test for the behavioural gate.
• The zero-shot baseline: no examples, no injected vectors. It echoes the input month, so the result is 0.083 (raised from 0.0 by the shift-by-0 operation).
• The injection setup: same zero-shot baseline, but this time we injected the extracted function vector into the hidden states.

The shift-by-k function vectors are extracted from six month-sets: all 12 considered, a 9-month set, two six-month ranges (January-June and July-December), and two four-month ranges (January-April, September-December). While the included months differ, everything else (extraction protocol, model, number of examples per prompt) remains the same. We obtain twelve shift-by-k vectors from each set that are verified by passing the following checks:

• The behavioural gate: whether the model can perform the task only from the examples shown in the few-shot prompts, so it is directed towards the model’s performance, not the vectors’. The results are compared to the 0.08 zero-shot baselines mentioned above.
• Stability: each vector is extracted twice from separate halves (50 prompts) of the prompt samples, and we measure how similar the two instances are (cosine similarity, where 1.0 means their direction is identical), and we obtained results above 0.98 for every pair. Interestingly, the restricted sets’ (where not all the 12 months are present) vectors are the most stable (0.991-0.993 versus 0.980 for the full set).
• Causal effect: whether injecting a vector in a zero-shot prompt improves the result. It does: even for the most broken vector, the model will be 3.3x more likely to output the correct answer, and overall accuracy similarly triples from 0.08 to 0.26.

Every single vector got verified and passed all these checks. We only had two close calls, both attributed to the non-range test sets (12- and 9-month sets).

After we injected the vectors, we looked at the actual outputs, specifically the distance around the circle between the output and the correct month.

After observing results that seemed to prefer a failure direction, we used a margin metric: if we get the correct shift-by-k month, we get one point for the “correct” category. If the output is just an adjacent month in either direction, regardless of k, we get one point for the “neighbor” category. Because shifts by 0, 1, or 11 have scoring biases (the intended output is close to or exactly the neighbor), we did not include them when we calculated the average margin. So this margin is the first category minus the second (across the nine qualified shifts, for all sets), and +1 means the vector always performs the shift-by-k task, 0 is an inconclusive, muddied result, and -1 means the output is always a neighbor.

The margins shown are for the Todd-style extraction (head averaging). The Hendel method runs, but it steers nothing when k is in the middle, so every set’s margin is negative, including the full one.

The vectors from the four range sets gravitate towards adjacent-month outputs. In the most extreme set, this happens for 95% of injected outputs (split almost evenly between a right and left neighbor), and cases where the output is off by 2-8 months from the queried month (not the correct shift-by-k one!) never occur. The vectors are not just bad at their task, they have a whole different identity instead: output the queried month’s neighbour.

But how could vectors failing their task so loudly have tripled the accuracy of the causal effect check? Well, 0.26 is an average over all months, and when k is 0, 1, or 11, the neighbour happens to be exactly or close to the correct answer. While the failure is clearly visible on a single shift, it becomes much harder to detect in the averaged result.

Even more interestingly, in the most extreme set, the September-December 4-month range, if we consider the September, October, November, December subset, the margin is exactly -1.000, so every single injection output was the neighbor. This also addresses the concern that the vectors were only copying the example input-output pairs and failing due to out-of-distribution issues. But exactly on the set with the lowest possibility of such failures (the subset of previously shown months, so a copying vector would work best), the vector is the most wrong in this fashion.

Could restricted sets provide too little information for the model to catch on to the task? But no, as in the few-shot setup, the model genuinely generalizes the shift-by-k task, and the sets with fewer months actually get better accuracies. This cannot be attributed to copying the example pairs, as accuracy is computed on held-out input months. Moreover, we re-ran the first check with months that did not appear in the prompt example samples at all, and accuracy did not drop.

A sweep across layers, at different strengths, tested whether some injection settings could rescue the broken vectors. None of them did: no positive margins across the board (0.00 is the highest number reported). The model is fine, the mystery box just fooled us.

Who's at fault?

Right off the bat, the anomaly’s main suspects are the number and restrictions of the examples. However, each set has the same number of examples per prompt. Regarding restrictions, the 9-month set still obtains a positive result (+0.34). “Fewer months” can’t be the sole culprit.

The culprit is actually the number of distinct months that appear in a prompt’s examples. A prompt from the full set has about 8.6 different months on average, one from the 9-month set has 7.1, the six-month ranges have 4.9, and the four-month ranges have 3.0, as examples are repeated. For each instance, the only thing that changes is the diversity of examples for the task. Dropping the month diversity raises the result of the first check: the few-shot setup prompt becomes easier to accomplish, as the score of the weakest shift rises from 0.38 to 0.83, because fewer months means fewer ways to be wrong. These measurements move in opposite directions, as quality certification rises while the vector strays further from the intended identity. This is why the first check’s two close calls were both on the least corrupted sets.​

The margin uniformly drops alongside diversity. When the set has at most 6 months, the extracted vector doesn’t just degrade, it changes identity, an identity that gets stronger with more restriction. This neighbor task vector is a blend of previous-item and next-item tasks, which are included in the classic function vector operations. This could be because, for example, a prompt with 9 examples but 3 distinct inputs underdetermines the task, and the neighbor one is the closest well-known alternative.

All this is measured on a single model, a single circular task family, using Todd’s method of extraction (we can’t compare it to the Hendel method, as no vector performing the intended task was produced). Whether the diversity phenomenon occurs in other classic task families with limited answer sets, and examples including only elements of certain restricted subsets, is a natural follow-up and the next registered experiment.

Why nobody opens the box (and how to do it)

Let’s go over the flaws of each check and how the vector identity failure bypassed them:

• Because the behavioral gate only checks the model, it tells us nothing about the nature of the vector (and it is intended to behave this way)
• Reliability doesn’t account for the identity of the object, only whether the two instances of extraction are similar, so we test whether we extract the same object every time, not whether it is the correct one.
• Averaging discards valuable information, and this bit us, as we already saw. We question whether the injection raises the correct answer’s probability, or the accuracy, over a multitude of shifts (it did, even for the most broken vector), so anomalies can go under the radar.

Each check has its own purpose: competence, stability, effectiveness, and none of them is identity.

The first and third checks were presented in Todd et al.’s paper (which also included the previous-item and next-item task vectors). Hendel et al.’s paper included a sort of reliability check by varying the examples shown in the prompts, but they were always selected from the entire full set. Makelov et al. investigate the intervention illusion, a similar phenomenon which shows that a successful activation patch does not mean we found the location of a concept in the model. Tan et al. also found steering vectors that fail to steer. Lin et al. show that the field is frequently overconfident in checks that don’t directly verify the base claim. However, our article turned out to be an empirical study of this diversity phenomenon, which is not included in their paper. While unreliability (“it doesn’t work sometimes”) is a recurring notion in the mechanistic interpretability field, this failure is of the “it works, but for the incorrect task” type and bypasses the verifications that combat unreliability.

Even though diversity can be computed in a single line of code, it goes unreported. However, strategies tied to it (for example, considering input subsets or restricting output values) are employed in the field, so diversity as a metric is clearly relevant.

To uncover imposters, identify the most likely separate function that masquerades as the intended one, and compute the margin: the difference between the number of occurrences of the intended output and the one attributed to the imposter, counted only in cases where they differ. Because injections are already performed to check their effect, this metric doesn’t require any additional run. We considered this metric only after directly looking at the injected outputs and discovering imposters that had slipped through.

Drawing the line

To validate our evaluation tools, we needed thresholds. To determine the chance/random output bar, we considered two approaches. The standard way is to generate random outputs with the correct shape, run each test on 300 such selections, and draw the line at the highest score. Another way is to shuffle the real, extracted vectors, run the same tests on them (only the “k” label of the shift-by-k operation changes), and draw a new line.

These two lines should be close, but their difference was baffling: early tests showed that the one from the shuffled vectors was 100 times the first one. So we calculated the median of their ratio across all tests, and it was 0.79. The ratio is not similar across cases, though. For two, the random line is 145-156 times too low, for one, it is 100 times too high compared to the second one, and they agree on the rest. The method for picking the qualification threshold influences a paper’s final result, and this should never be the case.

What's left in the box

This was run on one model, one circular task family, and one extraction method. The natural, registered follow-up experiment is to run the same test on another classic task family with a limited answer set to see whether the diversity phenomenon is general.

As extracted vectors are increasingly used in real settings, we need to be certain of their identity. This article shows an approach to make them more trustworthy by catching imposters.

The code, numbers, and decisions are available in this repository, from which these experiments and runs can be reproduced: https://github.com/star2vec/tarcle.

The box? Shake it, weigh it up, run every check you would like: just make sure to open the lid and confirm what’s inside.