Evidence for feature-specific error correction in LLMs

LLMs are commonly assumed to use superposition to represent more features than they have dimensions. The evidence for this is mostly indirect — chiefly the success of SAEs at extracting interpretable directions. A stronger claim is that models also compute in superposition, and for that we have only theoretical evidence.Hänni et al. 2024 showed that computing in superposition requires error correction: because features are embedded non-orthogonally, each active feature produces a small interference activation along the others, and a network has to suppress this noise while preserving feature signal. This means the error correction has to treat feature directions differently from generic ones — it has to be less sensitive to perturbations along non-feature directions than along feature directions. We call this feature-specific error correction (FSEC). It’s a prediction of the computation in superposition picture, and it hadn’t been tested empirically. This post proposes a test and runs it. The method uses activation plateaus: in-distribution activations are robust to small perturbations, and that robustness is direction-dependent. We perturb the residual stream and measure whether the model is more sensitive to a candidate feature direction than to a mixture of two such directions. We summarize this with an norm: at the response is a quadratic form in which no direction is privileged over its mixtures (and which can’t privilege the many directions superposition requires), while means the pure directions are privileged, as FSEC predicts. We measure for contrastive directions and for MELBO and SAE directions, against for PCA, random, and random-difference controls. The contrastive result replicates across six model families, and we reproduce the effect in a toy model with known ground-truth features. These results indicate that candidate feature directions (contrastive, SAE latents, and MELBO) are (approximations of) directions treated preferentially by the network.

Key results • We propose FSEC as a test of computation in superposition: we model the response to a perturbation as a function of the norm of its decomposition into candidate feature directions, where FSEC predicts . • We find for three types of candidate feature direction — contrastive, MELBO, and SAE-decoder — with the contrastive result replicating across six LLMs, while PCA and random baselines give . • We show FSEC also occurs in a toy model of error correction with ground-truth features, where degrades toward 2 as directions are rotated away from the true features.

Why we think this matters First, to the best of our knowledge, this is the first empirical evidence in non-toy models (we use LLMs) for FSEC, and in turn for computation in superposition. We cannot directly show that computation in superposition happens. However, if it does, we believe it requires FSEC, and we do find empirical evidence of FSEC. Second, appears to separate feature directions () from non-feature directions (), and measuring it requires only a generic input to perturb rather than labelled data. The natural idea is to use this as an unsupervised feature-finding objective — searching for directions that maximize . While we were excited about this, early results have updated us away from it working: appears to be necessary but not sufficient, and maximizing it surfaces high- directions that don’t otherwise behave like features. The rest of this post is copied from thepaper, omitting the abstract and the list of contributions. Code is available at github.com/FranciscoHS/fsec-paper.

Introduction Representations in large language models (LLMs) are poorly understood. It is commonly assumed that LLMs make use of superposition (Elhage et al. 2022) to represent more concepts than they have dimensions available, and potentially to compute in superposition (CiS) (Hänni et al. 2024, Adler & Shavit 2024, Olah et al. 2025). However, we have only indirect evidence for superposition, chiefly the success of sparse autoencoders (Cunningham et al. 2023, Gao et al. 2024, Templeton et al. 2026) at extracting interpretable directions, and only theoretical evidence for CiS (Hänni et al. 2024). An empirical prediction of CiS is that neural networks must correct interference noise while preserving feature signal (Hänni et al. 2024). This requires networks to be less sensitive to perturbations along non-feature directions than along feature directions. We name this property feature-specific error correction (FSEC). While we cannot rule FSEC out or in without ground-truth feature directions, we can still ask whether FSEC-like behaviour occurs for any directions: if the model’s error correction privileges certain directions, those directions are candidate features, and we can detect them by measuring sensitivity, requiring only a generic input to perturb rather than feature-specific labeled data. We provide the first empirical evidence of FSEC, showing that the robustness of LLM activations to perturbations privileges certain candidate feature directions over others. Concretely, we perturb residual-stream activations at early layers and measure the downstream response as a function of perturbation direction and magnitude. We construct candidate feature directions via contrastive means for a variety of concepts, including languages, programming languages, gender, sentiment, registers, and verb tenses. Across Gemma-2-9B (Gemma Team 2024), Qwen3-1.7B (Yang et al. 2025), Llama-3.1-8B (Grattafiori et al. 2024), Mistral-7B-v0.3 (Jiang et al. 2023), Aya-Expanse-8B (Dang et al. 2024), and Yi-1.5-9B (Young et al. 2024), we find that contrastive feature directions elicit a stronger downstream response than mixtures thereof, consistent with FSEC that privileges feature directions while suppressing interference along non-feature directions. We formalize this by modeling the LLMs’ response to perturbation as an norm of the perturbation’s decomposition into candidate feature directions. The case reduces to a basis-invariant quadratic form, meaning no choice of basis is privileged; breaks this invariance, indicating that the candidate pure feature directions are more sensitive than their mixtures, as predicted by FSEC. We measure for contrastive directions across models. MELBO (Mack & Turner 2024a) and SAE directions, which also aim to recover model features, likewise yield , albeit with smaller values. PCA and random directions do not, consistent with the interpretation that reflects alignment with the model’s features. We validate this methodology in a toy model of error correction with ground-truth features (Section 5), confirming that degrades toward as directions are misaligned with the true features.

Related Work Activation plateaus. Prior work has established that in-distribution activations of LLMs are resistant to perturbations (Heimersheim & Mendel 2024, Janiak et al. 2024, Shinkle & Heimersheim 2025), a phenomenon known as activation plateaus. We introduce a novel measurement of the activation plateau boundary geometry, and identify its connection to FSEC. Direction-dependent sensitivity for feature finding. Prior work has exploited the fact that LLMs have direction-dependent sensitivity to perform unsupervised optimizations for directions maximizing downstream response, resulting in interpretable steering vectors, including MELBO (Mack & Turner 2024a, 2024b). We make use of the same phenomenon to empirically probe error correction, and apply our analysis to directions obtained in this way (Section 4.3). Unlike this line of work, our analysis adds the novel study of the sensitivity geometry and its connection to error correction. Error correction for computation in superposition.Hänni et al. 2024 argue theoretically that computation in superposition requires error correction that privileges feature directions. We provide empirical evidence in favor of superposition and error correction occurring in LLMs.

Methodology We probe error correction in LLMs by perturbing residual stream activations and measuring the downstream response. In-distribution activations are robust to small perturbations, a phenomenon known as activation plateaus (Heimersheim & Mendel 2024, Shinkle & Heimersheim 2025). This robustness is direction-dependent, i.e., the model is more sensitive to perturbations along some directions than others. Two lines of evidence suggest feature directions in particular are privileged: empirically, prior work recovers interpretable directions by optimizing for sensitivity, the inverse of robustness (Mack & Turner 2024a, 2024b); and theoretically, FSEC predicts that feature directions are privileged. We test this by comparing the downstream response along candidate feature directions to the response along non-feature baselines. In all experiments, we perturb the residual stream at an early layer (default ) and measure the downstream response at the second-to-last layer, maximizing the number of intervening layers. This is because activation plateaus are known to be more pronounced the greater the distance between perturbation and measurement (Shinkle & Heimersheim 2025). We avoid the last residual stream layer, which is known to behave atypically. The downstream response is computed by patching the perturbed activation back into the model (Meng et al. 2022, Heimersheim & Nanda 2024), performing a forward pass, and taking the distance between the perturbed and unperturbed residual streams at the measurement layer. We show in Section 4.3 that our results are robust to varying both the perturbation and measurement layers, and hold also when measuring cosine distance or KL-divergence in the logits. To distinguish directional effects from those of magnitude, we follow prior perturbation analyses of activation plateaus (Heimersheim & Mendel 2024) and perturb by rotating the activation vector towards the perturbation direction while keeping the activation’s norm constant. We refer to this as a norm-matched perturbation. Concretely, a perturbation of angle of toward is: where is the component of that is orthogonal to . We typically perturb at the last token position, but show in Section 4.3 that our results are robust to this choice. We quantify the model’s sensitivity along a given direction as the plateau-breaking angle, the smallest perturbation angle for which the downstream response exceeds a threshold . We set per direction-pair, at a level that is guaranteed to be crossed by both single-axis sweeps.[1]

We aggregate the downstream response across a fixed set of inputs (anchors) —last-token residual-stream activations of 5-token FineWeb prompts (Penedo et al. 2024)—by taking the median over anchors. Writing for the response of anchor when perturbed by angle along direction , the single-axis response curve is

is then the largest median response attained when sweeping along alone. For a pair we set the threshold

a fraction of the smaller of the two single-axis maxima. Taking the smaller maximum ensures both axes reach , so the single-axis plateau-breaking angles that calibrate the fit are always defined. Throughout we use . We show in Section 4.3 that our results are robust to varying within half-to-double its nominal value. We perform perturbations along six types of directions: contrastive (Panickssery et al. 2023, Turner et al. 2023), MELBO (Mack & Turner 2024a), SAE latents (Lieberum et al. 2024), PCA directions, random, and random-difference directions. The first three are all candidate feature directions, while PCA, random, and random-difference directions function as non-feature baselines. PCA directions are computed by performing PCA on the residual-stream activations arising from a randomly selected sample of 10000 5-token-long FineWeb inputs (Penedo et al. 2024). Random directions are sampled isotropically on the unit sphere, . Contrastive directions are constructed as the difference of mean activations between two sets of matched prompt pairs differing along a single concept (e.g., for gender, “He ran home” is paired with “She ran home”) (Panickssery et al. 2023, Turner et al. 2023). The prompts are LLM generated and human verified, and are available in our code release. Letting denote the activation at the perturbation layer for prompt , Random-difference directions apply this same construction to randomly paired activations rather than concept-matched ones, giving a control that is matched to contrastive directions in everything except semantic content. We average differences between pairs drawn at random from the same 10000-input FineWeb sample used for PCA, and normalize: Unlike contrastive directions, the paired prompts share no concept, so we do not expect them to be features. Unlike isotropic random directions, consist of differences of activations, and hence inherit their covariance. This makes it our most stringent non-feature baseline: it differs from a contrastive direction only in the absence of concept-coherent pairing. We construct 40 such directions. MELBO directions (Mack & Turner 2024a) are constructed by optimising a unit-norm direction at the perturbation layer to maximise the distance between perturbed and unperturbed activations at a downstream layer. Unlike contrastive directions, this requires no labelled prompt pairs as the directions are found unsupervised, and have been shown to produce interpretable steering when applied at sufficient magnitude. SAE latents are decoder columns of a sparse autoencoder trained on the model’s residual stream activations (Bricken et al. 2023, Cunningham et al. 2023). Each column is a candidate feature direction under the SAE’s decomposition. We select the top-33 most active SAE latents in the same sample of 10000 FineWeb inputs used for PCA directions. We do this for Gemma-2-9B only and use Gemma Scope (Lieberum et al. 2024) (the width-16k residual-stream SAE at layer 2). We also perform perturbations toward combinations of two directions . We first restrict them to the tangent space at via the projector , and orthonormalize (Gram–Schmidt) to obtain unit vectors satisfying and . We then form the unit tangent direction which interpolates between at and at . Perturbing by angle toward then proceeds exactly as in the single-direction case (Equation 1), We observe behavior (see Section 4) that appears consistent with the downstream response depending on the perturbation’s projections onto privileged directions, raised to a common power . To formalize this, we model the downstream response to a perturbation vector as where are unit directions, the weights are free parameters capturing each direction’s sensitivity, and is a scalar function. When we perturb along two directions simultaneously and vary their relative weighting via the mixing angle , the plateau-breaking angle becomes a function of ; plotted in suitable coordinates, this function traces out a superellipse (see Figure 2). Taking and , the inner sum collapses to two terms provided the remaining contribute negligibly along . This is expected under superposition, where features are packed nearly orthogonally (Elhage et al. 2022), so the projections of onto other directions are small; the quality of the superellipse fits in Section 4 confirms the approximation. For the norm-matched perturbation of Equation 9, the displacement is . The term lies along and is therefore orthogonal to each , so it drops out of every projection, ; only the tangential component contributes, so The plateau breaks when ’s argument crosses a fixed level (the value of the argument at which the response reaches the threshold ), giving the implicit equation for the plateau-breaking angle : The single-axis sweeps (at and ) calibrate the weights at this same threshold: gives , and gives , where are the plateau-breaking angles measured along and individually. We calibrate on the single-axis sweeps because each isolates one weight; this is a choice of convenience rather than a requirement. Substituting and dividing through by eliminates the unknown threshold and the activation norm, yielding This means that the plateau-breaking angles define a superellipse of exponent in the normalised coordinates . We estimate by minimizing the squared residual of this equation over the measured points.

Results We begin by illustrating our measurement procedure on a concrete example. Figure 1: Measuring plateau-breaking angles. Downstream response as a function of perturbation angle for two contrastive directions (Wealth, Gender) and an equal combination of both, at Gemma-2-9B layer 2, illustrating how plateau-breaking angles are extracted. The plateau-breaking angle is the angle at which the downstream distance first exceeds the threshold (here ). The grey dashed “Random” curve shows the median response across perturbations along 10 isotropic random unit directions. All curves are medians over 30 FineWeb anchor prompts; we omit uncertainty bands because the absolute scale varies substantially from anchor to anchor in a way that is shared across all curves and largely cancels in within-anchor comparisons. The within-anchor gap between each feature direction and the random baseline is nonetheless robust: at every angle, the feature curve exceeds the random baseline for most anchors (), and the median feature-minus-random difference is significantly positive (its 95% bootstrap confidence interval over anchors excludes zero). Figure 2: Iso-plateau boundary. Plateau-breaking angles for the Wealth Gender pair at Gemma-2-9B layer 2 (per-pair threshold ). The superellipse exponent is fit in the normalised coordinates of Section 3. The boundary is well fit by a superellipse of exponent (fit residual ); indicates these directions are privileged. Figure 1 shows the downstream response in Gemma-2-9B when perturbing along the Wealth contrastive direction, the Gender contrastive direction, and an equal combination of the two. The vertical dashed lines mark the plateau-breaking angle for each of the directions. Repeating this measurement across a range of mixing angles yields a plateau-breaking angle for each, which we plot in Figure 2. The resulting boundary is well fit by a superellipse with exponent . To verify that combining contrastive directions in this way is meaningful, we show in Figure 3 that steering along these combinations produces interpretable compositional behaviour changes. Figure 3: Compositional steering at Gemma-2-9B layer 2. Sample completions for the prompt “The other day I met someone who” under no steering, Poverty alone, Female alone, and the Poverty Female composite. Steering uses the contrastive Wealth and Gender directions (Appendix A); each row shows a single pole—the low-wealth (poverty) pole of Wealth and the feminine pole of Gender. Highlighted spans: orange for poverty and blue for feminine words. Concretely, steering along the Wealth direction (toward its low-wealth pole) produces poverty-themed completions from a neutral input, steering along the Gender direction produces feminine-coded completions, and steering along a combination of the two produces completions that are both poverty-themed and feminine-coded. This supports the reasonableness of the combination-perturbation setup, though we note that interpretable steering alone does not establish that a direction corresponds to a model feature.

Superellipse exponents across direction types We now systematically characterise these patterns across directions, direction types, and models. The superellipse exponent has a simple geometric meaning. Perturbing along a single direction breaks the plateau at some angle; perturbing along a mixture of two directions may break it sooner, later, or at the same point, and which of these occurs is exactly what records. If the model responds only to the overall size of a perturbation (its norm), then splitting it evenly between two directions is no different from concentrating it on one, and the plateau-breaking angles trace an ellipse (). If instead the model is specially sensitive to the individual directions—so that a perturbation matters only insofar as it aligns with one of them—then an even mixture, which aligns only partially with each, must be pushed further before the plateau breaks. The boundary then bulges outward, reaching a square () in the limit where each direction has a fully independent threshold and the response fires only as the projection onto either one crosses it. The exponent measures how far toward this “independent thresholds” regime the model sits. Each axis is normalised by its own plateau-breaking angle so that only the privileging of the pure directions over their mixtures—not their individual sensitivities—affects . This selectivity is exactly what error correction requires: staying responsive to individual features while suppressing generic mixtures of them. We give the formal statement—that cannot privilege feature directions under superposition whereas can—in Section 6.1. We repeat the analysis across six direction types. For contrastive directions, we form 318 pairs from 33 directions (14 binary semantic concepts, 10 natural languages, 9 programming languages), keeping only pairs with intra-pair cosine similarity below 0.1. The filter is needed because our two-direction analysis orthogonalises the pair (Section 3): if the two directions already have significant overlap, orthogonalising one against the other distorts it substantially, so the perturbation no longer probes the labelled feature. The conservative 0.1 threshold keeps this distortion small. The full list and pairwise overlap are in Appendix A. For MELBO, we use 528 pairs obtained via the procedure described in Section 3. For SAE latents, we use top-activating decoder columns from Gemma Scope’s residual stream layer 2 width-16k SAE. PCA and random directions serve as non-feature baselines, as do random-difference directions, for which we form 780 pairs from 40 directions and keep the 710 with intra-pair cosine similarity below 0.1. Figure 4: Superellipse exponents by direction type. Each dot is one fitted superellipse exponent for a pair of directions. The black horizontal lines are the per-column medians. The dashed orange line corresponds to the isotropic reference. The white markers are the per-column means, with error bars corresponding to the 95% confidence interval on the mean, computed by direction bootstrapping. Candidate feature directions (contrastive, MELBO, top-activating SAE latents) sit consistently above . The PCA, random, and random-difference baselines cluster at . The superellipse fits are good: no per-condition median fit residual exceeds . We observe that contrastive directions (mean 2.42, 95% CI [2.29, 2.62], median 2.30), MELBO directions (mean 2.21, 95% CI [2.01, 2.42], median 2.16), and top-activating SAE latents (mean 2.28, 95% CI [2.14, 2.47], median 2.19) sit consistently above . This indicates that the model is more sensitive to perturbations along these directions than along their mixtures, consistent with them being features and with FSEC. We emphasize that our result is not merely that these directions are sensitive—as prior work on activation plateaus and MELBO has shown—but that they are more sensitive than mixtures of them, which is precisely what measures. The same is not true for any of our baselines: top PCA directions (mean 2.03, 95% CI [1.94, 2.12], median 1.97), random-difference directions (mean 2.05, 95% CI [2.00, 2.10], median 2.02), and random directions (mean 2.03, 95% CI [2.00, 2.07], median 2.02) all cluster at . As discussed in Section 1, means that, in the per-axis-normalised coordinates, the plateau boundary is elliptical: mixtures break through at the same rescaled magnitude as the pure directions, so neither direction is privileged over its mixtures. This is inconsistent with the baseline directions being aligned with feature directions, under the assumption of FSEC.

Rotating away from feature directions We have shown that candidate feature directions have while controls do not. We now show that decays monotonically to as we rotate away from candidate feature directions. We take a pair of contrastive feature directions and we rotate each of the directions toward a fixed random orthogonal direction by an angle , re-orthonormalize the pair, and refit, so that recovers the original contrastive pair and yields a random direction. We sweep over a subsample of 40 o