a recurrent llm is quite easy to interpret but very hard to steer

TLDR;

Ouro-1.4b-thinking is broadly interpretable with logit lenses and linear probes. It's also steerable but does 'clean' foreign concepts out of the residual stream if they're injected before the last loop. This could have nasty implications for safety.

Code + data: https://github.com/mild-rgb/ouro-experiments + https://huggingface.co/datasets/mild-rgb/ouro-1.4b-thinking-evals

If you're not familiar with the Ouro family recurrent models, I recommend taking 5 minutes with your favourite AI agent to research them. This post may not make much sense if you don't

Intro/Structure

I evaluated Ouro-1,4b-thinking on 16 MBPP python tasks and 24 GSM8K questions. I recorded the residual stream at 4 layers (0, 6, 18, 24) per loop while the model was doing the questions. I then applied standard mech interp techniques to the residual stream recordings for the first two experiments. They broadly work as normal and gave some interesting results.

In my 3rd experiment, I try CAA on the model and intervene on each loop. I find that steering works much better on the last loop, and in some cases, not at all if not applied to the last loop. This is quite concerning because it raises the possibility of a misaligned recurrent model having several loops to plan around the consequences of being steered.

Experiment 1 - loop indexing:

Method

I then trained a 4-way logistic regression probe to predict which loop a standardised residual stream vector came from. I also trained a control probe that only sees the log of the vector norm as a control.

Results

Discussion

Loop index is definitely linearly represented as the linear probe does so much better than the norm-only probe. The worst results are at layer 18, which suggests that there's no universal loop counter. Instead, I think that a loop index is written in at the end of a loop and then removed at the start.

Experiment 2 - logit lens

Method

I applied the final norm and unembedding at the end of each loop on data taken from . This means that instead of one token prediction after four loops, one token prediction was shown instead. I then recorded the top-1 token and the logit entropy at each loop.

I then calculated the %age of tokens that were unchanged after each loop.

Results

Note: true spaces only really occur in code. Almost all words have spaces included in their tokens, so don't read too much into the spaces

Table of how tokens unchanged after loops

Table of how entropy changes after loops

Discussion

The model broadly seems to decide what it's going to write on the first loop. When it does change its mind after the first loop, it's between quite closely related words. 'is → becomes → becomes → now' is quite a good example of this pattern

Experiment 3 - CAA

Method

I did standard CAA with 6 concepts and extracted directions for each of the concepts. I measured the cosine similarity between concepts between layers. I then experimenting with steering at different loop indexes. I used random directions of equal norm as controls. I measured the effect of steering by subtracting the change in probability mass caused by a random control from the change in probability mass caused by a concept being injected.

Results

Cosine similarity between concepts is quite high across layers and in a few cases, equal to 1.

Steering at all positions and loops generally saturates at alpha values of 0.4. I also noticed a pattern where steering at the last loop is far more effective than earlier loops. I initially suspected that this was due the last loop having a larger residual norm but the effect survives even when I steer with the raw vector from layer 1.

Graph of effect vs injection loop

I also tracked the perturbation of the residual stream at each loop after injecting concepts at loop 1. I also injected matched random vectors as controls. I found that the residual stream stays perturbed when concepts are injected early but 'rotates' them away from the useful direction.

Graphs showing steering at layer 12 during loop 1 being neutralised + random controls

Discussion

Ouro has very similar linear representations of concepts between loops. This is shown by high cosine similarity between concepts extracted at different loops and the fact that steering vectors transfer between loops.

The increasing effect with later injections is really interesting. It suggests that Ouro has an 'awareness' of which concepts should be present in the residual stream and will seek to self-repair when they're injected.

Conclusion/Overall discussion

Ouro is broadly interpretable like a standard non-recursive LLM. The basic techniques work to decode its neuralese. Logit lens applied to the final layer of Ouro's intermediate loops is equivalent to reading a non-recurrent model's CoT.

The linear representation hypothesis seems to hold. Due to the high cosine similarity between concepts extracted from different loops, I believe that linear probes, j-lenses, and other interpretability tools trained on final loops would also transfer to intermediate loops quite well.

I believe that the inter-loop similarity is due to the early exit gate that was used in Ouro's training. This encourages easily decodable intermediate layers as the model is rewarded in training for producing the correct token early in the loop cycle. As there's only one unembedding matrix, intermediate loops have to remain similar to the final loop.

I'd also be quite curious to see an MoE recurrent model. Experts that only activate on certain loops would be quite interesting.

Steering is quite different from standard LLMs. Steering applied to early loops can be ineffective as the perturbation is rotated away from the target direction. This raises the possibility of models which could 'plan' around any steering and has quite bad implications for safety evals.