VPD's Evaluation Adversary Finds Systematic Defects in the Released Decomposition

TL;DR: adVersarial Parameter Decomposition (VPD) decomposes model weights into simple components and assigns labels that specify whether a component is needed for a given input. A core aspiration of VPD is robust mechanistic faithfulness - the property that switching off any set of unneeded components for a given input leaves the model's output approximately unchanged. The authors use an evaluation adversary to measure the degree of robustness, and report that the decomposition is not at all robust to very long adversarial searches. The authors list this as a limitation, but also argue that complete robustness is not necessarily the goal. Their discussion implies two conditions under which a lack of robustness at large step counts may not matter in practice: 1) the adversary could be finding defects in the decomposition that are tuned to particular inputs, rather than systematically carrying over to other inputs, or 2) the masks a practitioner might apply may be very different from adversarially found masks. To test the first condition, we ran a series of measurements on the paper's released decomposition of a four-layer, 67M-parameter language model and found that from 20 steps on, the faithfulness defects the evaluation adversary finds on one batch of inputs systematically transfer to other inputs never seen by the adversary. The adversary is therefore not simply exploiting interference noise that is particular to one set of inputs. We explore why the adversary used during training fails to prevent systematic defects, and we suggest a practical improvement to the adversarial training process that could mitigate these weaknesses.

Each blue dot represents one source found by the paper's evaluation adversary while searching a batch of 128 texts for one of five step counts (20, 40, 80, 160, or 320). A source is a set of per-component values the adversary selects, and it determines the mask used at every token of the batch. The y-value of each blue dot is the average induced KL divergence over 16 unseen batches of text. The average seen and unseen KL divergences are within 20% of each other across the whole range of 20 to 320 steps (blue dots hug the diagonal), showing that the adversary's discovered source is not just exploiting quirks of the batch it searched. The dashed line represents the paper's standard 20-step search applied directly to the unseen batches, the regime the paper calls "somewhat robust" (0.75 nats on our batches, while the paper reports 0.83 on its own batch). The orange dots serve as a control. Each is the average KL divergence over eight copies of a blue dot's source, each with its values shuffled at random among comparable components, applied to the same unseen batches and drawn at the corresponding blue dot's x-value. The orange dots demonstrate that output damage is due mostly to which components the adversary found, not how many it found.

This post is a follow-up to our first audit, which demonstrated that mechanistic faithfulness of the released decomposition can break down appreciably under the kinds of aggregations and deletions a practitioner might perform. In response, the VPD team re-ran that audit on a newer, unpublished decomposition of the same model and found it to be significantly improved, though still imperfect (see Lee Sharkey's comment). However, we audit only the decomposition released with the published VPD paper, because the newer decomposition is not publicly available. The code, result tables, and detailed specifications for how computations were performed in this post are available here.

Background

VPD's Adversaries

VPD decomposes a model's weights into simple components and, for each token of any input, provides labels between 0 and 1. These labels determine how much one can turn off each component without affecting the model's output. An ideal decomposition is mechanistically faithful, meaning one can switch off any set of unneeded components without changing the model's output. Earlier methods, such as Stochastic Parameter Decomposition, used random masking to foster mechanistic faithfulness. However, the authors found that random masking was insufficient on its own, so they added adversarial techniques to improve the process.

There are two types of adversaries used in the VPD paper - the training adversary and the evaluation adversary. The training adversary in VPD performs gradient search over the component masks (the multipliers for component outputs) in an attempt to maximize KL divergence from the original model output. It can only move each mask to a value between its causal importance label and 1. The training adversary sets one value per component at each token position of each text in the batch, carries those values from batch to batch, and updates them with only three small steps per batch. The evaluation adversary used by the paper is similar, with some important differences. Its search starts fresh for each batch of 128 texts, and it is allowed only one source value per component - that is, the value is shared by every token of every text in the batch. Then, for a given input, each token's mask is the shared source rescaled into the range between that token's label and 1. As a result, masks can still differ from token to token. The evaluation adversary is run for between 20 and 320 steps, and the authors say that the KL divergence this adversary inflicts is their primary indicator of faithfulness.

Systematic Defects Versus Interference Noise Manipulation

The sharing of the source among inputs in the batch is meant to encourage the evaluation adversary to find systematic defects. The paper defines systematic defects in its appendix on the adversarial reconstruction loss as cases where "a particular choice of ablation masks changes the model output on many data points even though it shouldn't." This contrasts with unimportant, noise-induced defects that are particular to only a small set of inputs. In the latter case, the authors use a toy model of superposition to explain how an adversary can exploit normally uncorrelated interference noise within the model. The proposed mechanism is that, on a given input, the adversary switches on the unneeded components whose interference noise moves the output one way and switches off those whose noise moves it the other way. By doing so, the adversary can manipulate noise that would normally cancel out across the components. Through this process, it can form a signal strong enough to affect model output significantly.

Even though the evaluation adversary was designed to find systematic defects, the paper does not report testing whether the discovered defects themselves are systematic. The authors note that their batch-shared source is "somewhat related to universal/shared adversarial perturbation methods that optimize one perturbation across many inputs" (Moosavi-Dezfooli et al., Shafahi et al., Mummadi et al.). For image classifiers, these types of perturbations are known to generalize - Moosavi-Dezfooli et al. found that a perturbation computed on a small set of training images fools the classifier on new images with high probability. However, it remains an open question whether the masks the VPD paper's adversary finds behave in a similar manner. This is because sharing a single source across a batch of inputs dilutes the chance of noise exploitation but does not fully remove it.

There are two routes to input-specific noise exploitation in the batch-shared setting. The first involves the adversary exploiting interference noise that happens to align among the 128 sampled texts but not among texts in general. This could come, for example, from tokens or topics that the batch shares at a higher rate than normal. The second involves the adversary sorting components by which way their noise, added up over the entire batch, points. To properly defend against these two routes, the authors themselves write that, ideally, they would use "the same sources for the whole data set." But this approach would be too computationally expensive to use in training.

Why the Defect Type Matters

In the VPD paper, the authors present a table of KL divergence as a function of the evaluation adversary's step count. The divergence grows from 0.83 nats at 20 steps to 40.2 nats at 320 steps, showing that the decomposition is not adversarially robust to large step counts. The authors acknowledge this limitation, writing in their section on limitations and future work that their decomposition is "not as adversarially robust as we would like" and that they "suspect that substantially higher levels of adversarial robustness are still achievable." However, they also note, immediately after presenting the results, that complete adversarial robustness "would not necessarily be desirable," and state in their Discussion that it is "unclear how much adversarial robustness is necessary." These statements, paired with the paper's picture of an adversary exploiting interference noise on particular inputs, suggest that some of the growth in KL divergence with step count may be acceptable. But this view is appropriate only if 1) the defects the adversary is finding are not systematic (that is, they do not carry over to other inputs), or 2) no masking a practitioner would actually perform is likely to coincide with these defects. The paper does not report an explicit test of either condition. This post focuses primarily on testing the first condition and only lightly explores the second.

Methods

Experimental Setup

To test whether the defects found by the evaluation adversary carry over to other inputs, we ran this adversary (using the authors' code) on the released decomposition. Our input texts were taken from the authors' validation split of the Pile (the same data the VPD paper evaluates on), and, like the paper, we arranged these in batches of 128 texts, each 512 tokens long. On each of 16 such batches, we ran five searches, each starting from a different random source, for a total of 80 searches on these batches. For each search, we recorded the source the adversary had found after 20, 40, 80, 160, and 320 steps. These are the step counts for which the paper's table reports adversarially induced KL divergence. We then applied each recorded source, unchanged, to 16 other, held-out batches of texts and measured the resulting KL divergence from the original model. Note that none of the adversarial searches ever saw these 16 other batches, and none of the held-out batches share any documents with the batch on which the source was found. Below, we give a schematic of this carry-over test:

A schematic of the carry-over test. The two texts stand in for batches - A for the 128 texts the adversary searched and B for texts the adversary never saw. A source is one number per component, shared by every token of every text in the batch, and the source found on A is applied to B unchanged. A source can take any value between 0 and 1. However, most of the values the adversary finds end up at exactly 0 or 1, so those are the only values we show. A source of 1 leaves a component fully on, while a source of 0 turns it down to the label assigned at each token of each text. So, on text A, the "Sky Fact" row is set to 0 and therefore keeps that text's labels of 0.7, 1.0, and 0.4, while on text B, the "Capital of" row, also set to 0, takes that text's labels of 0.0, 0.8, 0.6, and 0.0.

Measuring Transfer to Unseen Text

Throughout this post, we measure output damage in the same way the paper does - as the KL divergence between the next-token predictions of the original model and those of the masked model, averaged over all tokens. A few reference points are helpful for building a sense of scale. A random source, which is the starting point of every search, causes 0.22 nats of KL divergence before the adversary has taken a single step. At the other extreme, the VPD paper reports that zeroing every weight matrix in the model results in about 67 nats of KL divergence. Our main point of comparison, though, is the paper's own standard 20-step search performed directly on an unseen batch of texts. This results in 0.75 nats of KL divergence on our batches (the paper reports 0.83 on its own batch) - a level the authors describe as "somewhat robust." This level is drawn as the dashed line in the figure at the top of the post.

To quantify how much KL divergence remains when a source is carried over to unseen text, we divide the average KL divergence the source causes on the 16 unseen batches by the KL divergence it causes on the batch on which it was found. Note that, before dividing, we subtract the 0.22 nats that a random source already causes from both the numerator and the denominator. This is because the baseline divergence induced by a random source will carry over to any text whatsoever, and failing to remove it would therefore inflate the fraction of the damage that appears to transfer.

Shuffled Sources as a Control

A source that turns thousands of components down to their labels might raise the KL divergence simply because of the sheer number of components being turned down, rather than because of which particular components the adversary chose. To separate these two possibilities, we used a shuffled control. For each source, we made eight shuffled copies and applied them to the same unseen batches of texts. In each shuffled source copy, its values are randomly reassigned among components corresponding to the same weight matrices and are used nearly as frequently. A shuffled copy therefore turns down the same number of components as the original source, within the same weight matrices. If the shuffled copies cause as much KL divergence as the originals, then the output damage is tied simply to how many components are turned down. If they cause much less KL divergence, then the output damage is due to which components the adversary chose. These shuffled copies are represented by the orange dots in the figure at the top of the post.

Results

The Adversary's Output Damage Transfers to Unseen Inputs

In our main experiment, we find that most of the KL divergence the evaluation adversary induces on the batch it searches carries over to other batches of text unseen by the adversary (see the figure at the top of the post). If the adversary were just exploiting quirks of the particular batch on which it searched, we would expect the blue dots in the top figure to fall well below the diagonal. Instead, they hug the diagonal at every search length, all the way up into the tens of nats, indicating significant transfer.

When we count only the KL divergence above the 0.22 nats that a random source already causes (see the Methods section above), a source carried over to unseen batches keeps 78% of its divergence at 20 steps (95% interval: 68% to 88%), 82% at 40, 92% at 80, and about 95% at 160 and 320. In other words, everything particular to the batch on which the adversary searched, including the two noise routes described in the Background section, accounts for only about a fifth of the KL divergence at 20 steps and under a tenth of the KL divergence from 80 adversary steps on.

So, most of the KL divergence carries over to unseen texts, but one might ask about the scale of this output damage, as well as how pervasive it is across unseen inputs. As noted in the Methods section, the VPD paper's own 20-step search run directly on each of our unseen batches results in 0.75 nats of KL divergence. A source carried over from a 20-step search on some other batch causes a little less than this - 0.60 nats on average. But from 40 adversary search steps on, almost every carried-over source exceeds the 0.75-nat reference. Further, the output damage is not confined to just a few texts, which is the case the paper has in mind when it writes that the problems may "only apply to a few data points". Rather, the output damage is broadly distributed across unseen texts. At 40 steps, a typical pairing of a source with an unseen batch pushes 97% of the batch's 128 texts above the 0.75-nat reference. From 80 steps on, the typical pairing takes every one of the texts above this level. Based on these numbers and the VPD paper's own definition (a particular choice of masks that changes the model output on many data points even though it shouldn't), the adversary is finding systematic defects.

Which Components Are Turned Down Matters More Than How Many

Because a source turns thousands of components down to their labels, it is possible that the output damage is partly caused by the sheer number of components being turned down, regardless of which particular components are chosen. The shuffled source copies (the orange dots in the figure at the top of the post) serve as a check for exactly this possibility. At every search length, the shuffled copies cause only 0.30 to 0.32 nats of KL divergence on the unseen batches, which is barely above the 0.22 nats that a random source induces. In other words, the transferred output damage represented by the blue dots in the top figure is due to which components the adversary chose, not how many it chose. From here on, we sometimes refer to such a source as a "pattern," since each one is a particular choice of components to turn down.

We also find that the adversary is not rediscovering one defect over and over. Instead, it finds many different damaging patterns, each of which carries over to unseen text. Sources found from different random starts on different batches agree on which components to turn down only slightly more often than chance (an agreement beyond chance of 0.05). Further, no small part of a pattern is responsible for its damage. Turning down a random three quarters of the components in a pattern causes only 4% of its damage, and nearly all of the damage arrives with the last quarter (see Sections 4.3 and 4.4 of the repository's report). In other words, the released decomposition contains many systematic defects, not just one.

The Adversary Is Not Just Exploiting Interference Noise

Components in a decomposition that are marked as unneeded for a given input, but left on, can still change the output logits by a small amount due to interference noise. In the paper's toy model picture, these output contributions of unneeded components point in many different directions for any given input text, and therefore cancel out and do not affect the output significantly. However, an adversary searching that text for damaging arrangements of masks can exploit this noise - it can choose to switch off just the components whose noise contributions to the output all point the same way. This breaks the cancellation and can shift the output substantially.

The paper then suggests what might happen when such an adversary's pattern is applied to a different input. In that case, the direction of each component's contribution to the output may be reshuffled, and the same switched-off set may no longer line up in the same output direction. This would cause contributions to cancel once again, and would result in the adversary's defect being particular to the input on which it searched, not systematic. But suppose this premise does not hold - if a component tends to push the output in the same direction on every text, then a pattern the adversary picked on one text can still cause a significant change in output on other texts, making the defect systematic. We tested this premise directly by measuring the logit-space output contributions of components under the sources found in our experiments.

For any given input text, one can measure the net shift in the output logits that the adversary's pattern causes. So, for each of the 80 adversary sources found at 320 steps in this work, we switched off each component one by one on its own. We then recorded, as a plus or a minus sign, whether that individual component's contribution moved the output the same way or the opposite way compared to the net shift on that input. We recorded this sign twice for each component - once for the batch on which the adversary source was found, and once for a text the adversary never saw. The following figure schematically demonstrates our results in (d) and (e), with a comparison to the VPD paper's toy model picture shown in (a), (b), and (c).

A comparison of the paper's toy picture of interference noise with a schematic of what we found for the released decomposition. In contrast with (c), where a reshuffling of component contribution signs occurs, our measurements on the released decomposition indicate that the reshuffling effect is not strong enough to prevent the defect from transferring to new inputs (that is, the searched batch in (d) and the unseen text in (e) share component contribution signs much more often than chance alone would produce). Each arrow schematically shows the sign of a component's first-order contribution to the output when that component is turned on. Only the sign is measured, so every arrow is drawn at the toy's length, even though the real contributions differ in size by orders of magnitude. The net arrows in (d) and (e) show the pattern's measured shift of the output, which is 95% as large on the unseen text as on the searched batch. Note that the actual net contribution depends on interactions among the components, not just the first-order contributions drawn.

So, in the real decomposition, the output contribution signs mostly do not change between the seen and unseen text, which goes against the noise-canceling effect represented in the toy model. Weighted by the size of their contributions, 74% of the switched-off components keep their sign between the seen and unseen texts. For comparison, randomly reshuffled signs are consistent between the two only about 55% of the time. We observe the same for the sources found at 80 steps. Overall, the toy model picture of component outputs reshuffling and canceling out on different inputs does not apply to the released decomposition. One caveat worth pointing out is that we measured each component's contribution on its own, and these one-at-a-time contributions add up to only about a quarter of the pattern's total output shift at 320 steps (and about two fifths at 80 steps). So, this test checks the premise of the toy picture, but it does not uncover the full mechanism behind the output damage we observe.

Are the Systematic Defects of Practical Concern?

Recall the two conditions, implied by the paper's discussion, under which a lack of robustness to an adversary allowed many search steps may not matter in practice: 1) that the adversary is finding defects in the decomposition that are tuned to particular inputs, rather than systematically carrying over to other inputs, or 2) that practitioners are unlikely to apply maskings that look like adversarially found masks. The first condition fails by the paper's own criterion, as shown in the Results section above. The lack of robustness that the adversary finds is a property of the decomposition itself, not of the particular texts the adversary searched. So what about the second condition? Do these adversarially found defects have any relationship to the maskings a practitioner would actually apply?

In the VPD paper, the authors argue that a mask applied in practice is "very unlikely to be exactly tuned to random noise," but that, if the decomposition were systematically defective, "we might have a realistic chance of picking a mask that causes the edited model to behave differently than the target model on many inputs." Thus, by the paper's own reasoning, our result moves the second condition from very unlikely to a realistic chance of being problematic. The paper's standard for judging that chance is a practical one - what matters is "the ablations we end up performing in practice" - and on this point our data agree with the VPD paper. That is, the control comparison we ran with shuffled copies of the adversarial source showed that random ablations of a similar size and kind do not change the output much.

In practice, though, the maskings a practitioner applies are not chosen at random. The paper's own example of practical use is an edit to the model that, for every token of the input, switches off the components driving some specific behavior. This amounts to one chosen mask applied to many inputs, which has a similar shape to pooled explanations of two or more inputs. Our first post offers one piece of evidence on this point. There, we pooled explanations by fully switching on every component that any of 64 tokens from other texts needed, leaving the rest at each text's own labels. Averaged over 1,024 texts, this pooling caused 0.80 nats of KL divergence relative to the original model (0.34 of which comes from each text's own explanation), which is about as much as the 0.70 nats that the paper's 20-step adversary causes when run on those same texts. It should be emphasized that this pooled masking is not exactly the same as the adversary's pattern. The two are similar - fully on for some components and set at the causal importance label for the rest - but the pooled masking acts on a different and much smaller set of components (about 4,500 fully on, whereas the adversary leaves about 60% of the 38,912 components fully on). What the pooled masking does show is that a masking chosen by a practitioner reaches the level of damage caused by the paper's 20-step adversary without any search.

So, although this work does not provide conclusive evidence on whether these systematic defects are of practical concern, there is a decent chance that they point to a real problem in the released decomposition.

A Suggested Improvement to VPD Training

Why the Training Adversary Misses Systematic Defects

VPD trains its decomposition against an adversary for 400,000 batches, with that adversary's KL divergence included in the loss. Yet the evaluation adversary, starting from a random source, finds a pattern that significantly carries over to unseen text within 20 steps, and one that causes at least 5 nats of KL divergence in almost every search within 160 steps. So, why didn't the training adversary find these patterns so as to allow for their removal? We see two possible explanations: 1) the training adversary, with one value per position and only three small steps per batch, gets too little signal per step to find systematic defects, and 2) something the authors themselves worry about - that the decomposition's optimizer might "trap the adversarial optimizer in some local extremum at some point during training, rendering the adversarial loss useless".

To test the first explanation on its own, we took the authors' own training adversary from their code, at its released settings, and ran it against the released decomposition with nothing else updating. We held all other parts fixed in order to prevent the decomposition from either moving away from the adversary or trapping it. The only aspect of the adversary that we changed from the paper's was how widely each adversarial value is shared. We tried three settings: per position (matching the VPD paper), per document (every token of one text shares a value), and across the whole batch (how the evaluation adversary operates). We let each of these versions run for 1,000 batches of 64 texts that the adversary had never seen (two random starts each). We counted the adversary as having found a significantly damaging pattern if its current source values caused at least 5 nats of divergence averaged over two consecutive unseen batches.

We find that the adversary, when shared per position as in the paper, never found a significantly damaging pattern in 1,000 batches. The per-position adversary's divergence on fresh batches crept from 0.23 to 0.26 nats, which is approximately the KL divergence of a random source. By comparison, on the batch on which the adversary had just been optimized, the divergence reached 0.47. This implies that the per-position adversary mostly just tunes its pattern to the texts in front of it, which is the input-tuned kind of defect the paper sets aside. Shared per document, the same optimizer found a damaging pattern after 374 and 325 batches, and shared across the whole batch, after only 123 and 88 batches. So sharing per document, which is what the authors' newer code does, took about three times as many batches to find a damaging pattern as sharing across the whole batch. This suggests that pooling the gradient over more positions per step is what helps find systematically damaging source patterns. Further, too little signal per step could, on its own, explain why training never encountered these problematic patterns. We cannot rule out, however, that during real training the decomposition also moved away from its adversary or trapped it, since we ran only 1,000 batches compared to the real training's 400,000.

All in all, this points to a promising improvement - pooling the adversary's gradient over more positions per step. In fact, while writing this, we discovered that the authors' newer code, public in their repository as of September 2026 (though not described in the VPD paper), already shares each value across a document. However, the newer code also keeps a pool of persistent adversarial values at a higher learning rate, so our per-document version only somewhat resembles the authors' newer adversary. Further, we did not directly test the effectiveness of the improvement we propose. Our finding is consistent with Lee Sharkey's own diagnosis, in his response to our first post, that the procedure for finding strong adversarial ablations "isn't getting good enough coverage of ablation-space."

Limitations

All of this work was performed only on one released decomposition of one small language model, and says nothing about VPD in general. The authors have a newer decomposition of the same model that is more robust to the evaluation adversary, but we cannot test it yet because it is not publicly released. We tested only texts of the kind the paper evaluates on, so whether these patterns carry over to other kinds of text remains an open question. Further, the paper does not say which adversarial step size produced the rows of its adversarially induced KL divergence table corresponding to longer runs, and our searches cause more KL divergence per step than the paper's rows show (for example, 2.1 nats at 40 steps, whereas the paper's 40-step row reports 1.35 nats). Because of this, we compare directly only against the paper's 20-step KL divergence value.

Conclusion

We have shown that the VPD paper's evaluation adversary finds systematic defects in the released decomposition. If the evaluation adversary finds a damaging source pattern when searching on one set of texts, this pattern causes similar increases in KL divergence when applied to texts the adversary never searched. This is true for both short and long searches. Further, the components' contributions to the output logits mostly maintain their alignment with the general output shift across different texts, so the adversary is not simply exploiting input-specific interference noise.

All of this implies that the decomposition's lack of robustness to longer adversarial searches is a property of the decomposition itself rather than of the texts searched. The first condition for setting aside the adversary's large output damage - that the defects are tuned to particular inputs - does not hold for this decomposition. The second condition - that a practitioner's maskings are unlikely to land on these defects - holds for random maskings, but is untested for practitioner-chosen maskings in a more realistic setting.

One likely reason the training adversary did not defend against the systematic defects is that it gets too little signal per step. With one value per position and only three small steps per batch, the training adversary tunes itself to the texts at hand and never finds a pattern that carries over. However, the same optimizer, with its values shared per document or across the whole batch, finds such a pattern within a few hundred batches of inputs. We therefore suggest pooling the adversary's gradient over more positions per step.

Acknowledgments

Thanks to the VPD authors for releasing their decomposition, code, and training runs, without which this audit would have been much more difficult, and to Lee Sharkey and the team for their generous response to our first post and for re-running our experiments on their newer decomposition.

1. The components that the adversary leaves on keep their sign just as often - also at 74% - so this stability is not created by the adversary's selection. In other words, a component tends to push consistently with, or consistently against, the adversarial pattern's output shift on every text, rather than flipping from one text to the next. So the output contribution sign is, to a large extent, a property of the component itself.
2. The released checkpoint holds no adversarial values, so the per-position version started from scratch on our own texts. Further, the two starts of each version differ only in the adversary's starting values, so they are closer to one run than to two.