Unsupervised Feature Discovery via Simple Clustering

This article was submitted as part of Neel Nanda's MATS Application

Abstract. This article examines whether clustering can discover features. We define a feature as “a direction in activation space associated with a property that is both interpretable to humans and useful to the model”. Clustering is done on activations from Qwen2.5-7B-Instruct (layer 20) via recursive binary k-means clustering: the first level of the tree is obtained via flat k-means with k=64 (2^6), then each leaf is repeatedly split with k=2 until 2048 (2^11) leaves are obtained. Qualitative analysis shows interpretable clusters from the very first level (64 clusters) organized by lexical, syntactic, and semantic categories. Next, we run an "actual vs. decoy" distinguishability evaluation. An LLM judge can distinguish an actual cluster from a decoy cluster ≈97% of the time. The same judge can distinguish an actual SAE feature from a decoy one only ≈79% of the time. Lastly, we run an activation patching intervention by replacing the actual activation with a cluster centroid. Results show that, after patching, the model behaves as if the last token was the one correlated with the cluster centroid and continues the decoding accordingly, overriding the prompt’s natural continuation. Overall, the results provide compelling evidence that clustering discovered features. Motio (Figure 1) is the interface that allows to explore clusters (available at motio.ratiokinetics.com). The codebase for clustering, interface, and evaluation is available at github.com/ratiokinetics/motio. Training and eval data can be shared upon request at nrcbtz@gmail.com.

Figure 1: Motio interface

1. Discovering Features via Clustering

Elman (1990) trained a Simple Recurrent Network (SRN) for next-word prediction given a modest dataset of two- and three-word sentences. Next, he collected the activations for various inputs and ran a hierarchical cluster analysis. The results reveal that activations for noun inputs and for verb inputs form two distinct clusters. Verbs further split between those for which a direct object is required, optional, or absent. Nouns split between animates and inanimates.

Crucially, the model was never shown these categories. The model organized its internal representations to minimize the prediction error: activations for inputs that predict similar next words end up geometrically close to each other. And such organization happened to match lexical, syntactic, and semantic categories.

Modern mechanistic interpretability introduces the notion of a feature as “a direction in activation space associated with a property that is both useful to the model and interpretable to humans”. Sparse Autoencoders (SAEs) are the most popular technique to extract features in an unsupervised fashion. SAEs operate by decomposing model activations. Clustering, by contrast, aggregates activations that sit close to each other and represents each group by its centroid: the mean of the activations assigned to the group.

Can we discover features via clustering?

To answer this question, I study activations from Qwen2.5-7B-Instruct. The clustering is done via recursive binary k-means clustering: the first level of the tree is obtained via flat k-means with k=64 (2^6), then each leaf is repeatedly split with k=2 until 2048 (2^11) leaves are obtained.

As a dataset, we sample 50k documents from monology/pile-uncopyrighted. Documents below 512 tokens are dropped. We truncate all remaining documents after the 512th token. We then pass each document through Qwen and extract the residual-stream activations for each token (excluding position 0) at layer 20. This yields nearly 25M activations.

The training dataset is a random sample of 1M mean-centered and L2-normalized activations. The cluster centroids at the first level (k=64) are learned on the full training dataset. After that, each binary split uses only the activations already assigned to the parent.

Once training is complete, each mean-centered and L2-normalized activation from the full dataset is assigned to the nearest leaf at the last level of the tree. During assignment, each activation is given an ID that identifies which of the 2048 deepest-level clusters it belongs to. The ID is an integer (0-63) followed by 5 bits. The ID is a path that lets you recover an activation's assignment at higher clustering levels. ID 46.01110 identifies first-level cluster 46 followed by the splits 0-1-1-1-0.

Motio (Figure 1) is the interface that allows you to explore the resulting clusters. The design provides an intuitive way to observe and study the “semantic laws of motion” that an LLM might have discovered (Wolfram, 2023).

The interface is partitioned into two sides. The LHS shows the cluster pixel grid. The RHS shows the activation examples from the full dataset. In particular, given a query of cluster IDs, the activation examples are the documents that contain consecutive tokens (highlighted) whose activations belong to the queried clusters. The activation examples are ranked by similarity score: the geometric mean distance from the cluster centroid.

Via the interface, you can explore clusters vertically by moving through different levels of the tree (from level 6, corresponding to 2^6 clusters, to level 11, corresponding to 2^11 clusters), or horizontally by chaining clusters together. Once a pixel is selected on the grid, each next pixel's color indicates whether appending the corresponding cluster to the query would yield activation examples. The darker the pixel, the more activation examples the query produces.

Finally, the 8×8 grid at level 6 is not a 2D embedding of the 64 centroids. The clusters are laid down via quadtree placement according to their labels 0..63, assigned during the clustering in arbitrary order.

2. Qualitative Analysis

In this section, I manually observe activation examples for various cluster queries to establish, at least qualitatively, whether the activations tend to organize around meaningful, coherent, and human-interpretable categories.

We begin with a cross-sectional analysis: we study the activation examples for six different clusters at the first level of the tree (k=64). Figure 2 shows six clusters and their corresponding activation examples. Even at the first level of the tree, it is already possible to see that activations geometrically structure themselves around lexical categories (“adjectives”, “verbs”, “nouns”) and semantic categories (“place names”, “biomedical stuff”, “drug codes”)

Figure 2: Cross-sectional analysis of six clusters at level 6 (k=64).

Next, we perform a vertical analysis: we follow one cluster down its subtree. Figure 3 shows the splits, at level 8, of cluster 14 (previously labeled as “nouns”). We can identify four semantic subcategories of nouns: “objects” (cluster 14.01), “spaces” (14.00), “concepts” (14.11), and “groups” (14.10). The first bit reveals how the “nouns” cluster was split at level 7: “physical stuff” (14.0) vs “social/abstract things” (14.1). Overall, the hierarchical taxonomy through vertical movement is not always the rule: binary splits might mix the same tokens on both sides or collapse onto one word. For example, “verbs” (cluster 2) does not separate into meaningful classes at level 7: is/was/are/be sit on both children.

Lastly, we perform a horizontal analysis: we chain together various clusters at level 11. We first identify six clusters corresponding to “was/were” (2.01011), “constructing verbs” (55.01010), "inaugurative verbs” (46.01110), and the prepositions “in” (28.01010), “by” (40.10001), and “to” (80.00000). Figure 4 highlights, in red, the query corresponding to “was/were” (2.01011) + "inaugurative verbs” (46.01110) and “in” (28.01010). The activation examples corresponding to that query are exactly what you would expect. Similarly, coherent activation examples arise from all six possible combinations of 2.01011 + 55.01010/46.01110 + 28.01010/40.10001/80.00000.

This example is not cherry-picked: chains of clusters very often yield highly coherent activation examples. I invite the reader to play around with the interface by drawing chains of clusters and observe if any coherent pattern from the activation examples emerges.

Figure 3: Vertical analysis of cluster 14 at level 8 (k=256)

Figure 4: Horizontal analysis of the combination of various clusters at level 11 (k=2048).

One amusing additional finding is that several clusters, all originating from the same top-level cluster 28, match the preposition “in”. Cluster 28.01010, identified before, corresponds to “in after a verb”; cluster 28.00100 corresponds to “in within medical contexts”, while cluster 28.11100 corresponds to “in as the start of a sentence”. This finding shows that activations organize not only around lexical and semantic categories, but also around syntactic categories.

Before moving to the next section, it is important to highlight, at the risk of being obvious, that clusters’ labels are assigned post hoc and are never seen during training. Indeed, the clustering procedure is fully unsupervised: no lexical/syntactic/semantic category is provided during training, and any human-interpretable structure emerges solely from the geometry of the model’s activations.

Previously, we defined a feature as “a direction in activation space associated with a property that is both interpretable to humans and useful to the model”. Clusters’ centroids are directions in the activation space, but the results of the qualitative analysis are not enough to claim that they are “interpretable to humans” or “useful to the model“. In order to establish more rigorously whether clustering can recover features, we need to measure:

• Whether the activation examples for a given cluster follow a coherent, human-interpretable pattern
• Whether the identified clusters have a causal downstream effect on model behavior.

3. "Actual vs. Decoy" Distinguishability Eval

To measure the coherence of activation examples for a given cluster, we run an "actual vs. decoy" distinguishability evaluation. The eval proceeds as follows.

Configurations. Seven: six motio configs (one per tree level, 6–11) plus an SAE baseline for the same LLM model and same layer (Neuronpedia's qwen2.5-7b-it/20-matryoshka-65k)

Bin preparation. An activation example is an activating token plus its context window (up to 25 tokens before and after) that satisfies a requirement given a source. For motio, the source is a cluster ID and the requirement is that the activation corresponding to the activating token falls inside that cluster. Motio activation examples are ranked by similarity scores. For SAE, the source is an SAE feature, and the requirement is that the SAE feature's activation has strength > 0 for the activating token. SAE activation examples are ranked by activation score. For a given motio configuration, one bin per cluster ID is created. For the SAE configuration, one bin per SAE feature is created. Each bin keeps the top-20 deduplicated activation examples and saves them as a snippet, with the activating token wrapped in <<...>>. Bins with fewer than 20 activation examples are dropped, so the actual bin count can be lower.

Trials. Per config: 500 trials. Each trial samples 21 distinct bins: the actual set includes all 20 snippets from one bin; the decoy set includes one random snippet from each of the other 20. We shuffle the snippets within each set, then randomly assign the two sets to set_0/set_1.

Judging. An LLM judge (Gemini 2.5 Flash, temperature 0) receives both sets and is tasked with identifying the actual set by outputting an index (0 or 1). A trial is correct when the judge's output matches the index corresponding to the actual set. The coherence score for a config is #correct/500. A random judge scores 0.5, any score above that suggests that the configuration captures human-recognizable patterns.

Figure 5: Distinguishability eval results (accuracy per configuration)

The results (Figure 5) show that, for the motio config, an LLM can distinguish the actual sets from the decoy set with near-perfect accuracy. This result confirms the observations that emerged from the qualitative analysis. Surprisingly, accuracy is already near-perfect at the first tree level of clustering (k=64) and improves only slightly as we traverse the tree and increase the dictionary size.

What’s even more surprising is the comparison with the SAE config baseline. The SAE has 65,536 features, while motio config at level 6 forms 64 clusters. The dictionary size differs by 1024x. Therefore, you would expect that the finer SAE dictionary produces much narrower concepts, and narrow concepts should produce coherent activation examples. Nevertheless, the results reveal the opposite: actual sets from Motio clusters, even at level 6, can be distinguished from decoys more accurately than the actual sets from SAE features.

On the other hand, the results can be explained by differences in bins’ candidate depths. At level 6, there are ≈161k candidate activation examples per cluster (10.32M activations / 64 clusters), so the top-20 activation examples are the top 0.01% candidates per cluster. In contrast, SAE has ≈ 39 candidates per SAE feature (2.55M activations / 65,536 SAE features), so the top-20 are about 50% of the available candidates per SAE feature. Therefore, the SAE’s bins include the long tail of low-activation examples that might confuse the LLM judge. This hypothesis aligns with findings from Huben (2024): the long tail of activation examples for a given SAE feature is often uninterpretable.

4. Intervention via Activation Patching

To measure whether an identified cluster has a causal downstream effect on model behavior, we run an activation patching intervention.

Given a cluster, take tokens [0…pos] as the prompt and generate a continuation twice: once untouched (baseline) and once patched. In the patched run, we replace the residual-stream activation after block 20 at position pos with an activation that has the chosen cluster centroid’s direction and the same distance from the dataset mean as the original activation. We apply the patch once at the last token during prefill; the patched activation then persists through decoding via the KV cache.

Figure 6 shows the intervention for cluster 35.00000 (“org package”), whose activation examples are typically the token org inside Java import paths, and a prompt ending at import java. Untouched decoding continues in the java.* namespace (.io.InputStream, then more java imports). After patching, decoding instead continues in the org.* namespace (.jboss.modules.ModuleIdentifier, then org.junit.Test).

Figure 6: activation patching eval - org package

Figure 7 shows the intervention for cluster 61.01100 (“sentence final .”) and a prompt truncated mid-sentence “... was”. Untouched decoding continues the sentence. After patching, decoding instead outputs a capital letter, as if it were starting a new sentence.

Figure 7: activation patching eval - sentence final .

Both examples show the same behavior. After patching, the model behaves as if the last token was the one correlated with the cluster centroid and continues the decoding accordingly, overriding the prompt’s natural continuation.

5. Conclusions

We applied recursive binary clustering to activations from Qwen2.5-7B-Instruct (layer 20) and asked whether the clustering would discover features.

The qualitative analysis (Section 2) showed, anecdotally, that clusters are organized around meaningful, coherent, and human-interpretable concepts, resembling lexical, syntactic, or semantic categories.

The experiments in Sections 3 and 4 provide compelling evidence that clustering can recover features: activation examples for a given cluster follow a coherent, human-interpretable pattern ("actual vs. decoy" distinguishability evaluation) and clusters have a causal downstream effect on model behavior (intervention via activation patching).

The results align with Elman (1990), despite a far more complex model architecture.

To the best of my knowledge, there’s no prior research on LLM feature discovery via clustering. Neuronpedia doesn’t include any cluster-based feature discovery interface, and there’s no trace of such a technique used in the context of feature discovery in Learn Mech Interp. Lastly, research via Perplexity suggests that the closest results come from Rumbelow (2026), albeit using a more intricate algorithm. This is surprising given the technique's simplicity and computational affordability.

To further corroborate the hypothesis that features can be discovered via clustering, further evaluations are needed. Fortunately, a large suite of SAE evaluation frameworks (SAEBench from Karvonen et al., Huben (2024): 2025 and RAVEL from Huang et al., 2024) can be repurposed to evaluate features discovered via clustering. Additionally, it is necessary to evaluate whether clustering escapes the limitations observed in SAEs, such as instability across seeds (Paulo and Belrose, 2025) and indifference to randomly initialized LLMs (Heap et al., 2025). Labels can be assigned to clusters via traditional autointerpretability (Bills et al., 2023) or via Natural Language Autoencoders (Kit et al., 2026).

Lastly, it would be interesting to study the trajectories activations follow between states (i.e., between following tokens) to examine whether any semantic laws of motion exist (Wolfram, 2023). Elman (1990) suggests studying what sort of attractors develop in transitions between states by carrying out a principal component analysis of the activation-pattern time series, and then constructing phase-state portraits of the most significant principal components.

References

Bills, Steven, et al. "Language models can explain neurons in language models." 2023.

Elman, Jeffrey L. "Finding structure in time." 1990.

Fraser-Taliente, Kit, et al. "Natural language autoencoders produce unsupervised explanations of LLM activations." 2026.

Heap, Thomas, et al. "Sparse autoencoders can interpret randomly initialized transformers." 2025.

Huang, Jing, et al. "RAVEL: Evaluating interpretability methods on disentangling language model representations." 2024.

Huben, Robert. (username: Robert_AIZI) “Comments on Anthropic's Scaling Monosemanticity.” 2024

Karvonen, Adam, et al. "SAEBench: A comprehensive benchmark for sparse autoencoders in language model interpretability." 2025.

Paulo, Gonçalo, and Nora Belrose. "Sparse autoencoders trained on the same data learn different features." 2025.

Rumbelow, Jessica. "Exemplar Partitioning for Mechanistic Interpretability." 2026.

Wolfram, Stephen. “What Is ChatGPT Doing:... and Why Does It Work?” 2023.

1. The definition modifies the one from learnmechinterp.com by adding the “human-interpretable” piece
2. Calculated via Euclidean distance