SONI: Selective Orthogonalisation via Noise Injection

This project was completed as a capstone for TARA. All code is available in github.

TL;DR

The Problem: Neural networks use superposition to pack many concepts into small latent spaces by making feature vectors almost-orthogonal. This entanglement makes models opaque and breaks safety interventions (e.g. concept erasure, activation steering) which rely on clean, isolated concept directions.

The Gap: Full orthogonalisation (via sparsity penalties) destroys model capacity, while Sparse Autoencoders (SAEs) only view the features without changing the underlying model geometry. We need a way to selectively orthogonalise specific directions.

The Solution: We introduce SONI (Selective Orthogonalisation via Noise Injection), a fine-tuning regime that uses targeted noise injection to selectively orthogonalise a chosen direction in the latent space. This requires no loss function modifications and preserves overall model performance.

The Results: We demonstrate on Anthropic's Toy Models of Superposition (TMS) that this method significantly increases the orthogonality of all other features relative to a target feature across varying dimensionalities, without completely forcing perfect orthogonality. While limited by co-activation failure rates at lower sparsities, in sparse regimes it provides a geometric guarantee that could make downstream safety interventions significantly more reliable.

Introduction

Superposition is a structural property of neural networks where many more concepts are represented than there are dimensions in the latent space. The model achieves this by compressing concepts into almost-orthogonal feature vectors. While this allows for efficient, dense representations, it creates a fundamental barrier for AI safety.

When concepts are entangled via superposition, the internal representations of the model become opaque and difficult to control. Many critical mechanistic interpretability and alignment methods—such as concept erasure, activation steering…