No sign of backtracking in latent reasoning: the final answer simply settles in instead

Solving a hard math problem is not linear, it's trial and error.

You drop an idea, pick up an earlier one, go back to a computation from another approach, until something clicks. You might scribble on paper, but even if you don't, you still remember the path to get to the result, as well as the other methods you tried before one worked.

Do language models do this too?

In chain-of-thought reasoning, we can see that they do: the backtracking shows up in the transcript, signaled by tokens such as "wait". But in latent reasoning, models think without writing tokens down, and the intermediate steps are just vectors. Recent work (Cui & Ye) implies that latent models backtrack as well. Do they?

Why would it even matter? If a latent model goes back to its earlier thoughts, those thoughts become a trace that we can monitor and potentially steer.

On the models we could evaluate (including the one the claim was made on), nothing resembled a change of mind, and what could seem like backtracking was actually just the final answer settling in. We also show some observations we made along the way, as well as why this question is harder to answer than it may seem at first glance.

What could latent backtracking be and how we defined it

There are two kinds of reasoning in LMs:

• Text: the model writes its steps as a sequence of tokens, so the backtracking can be pinpointed (“wait, that’s wrong”, “wait, actually”) and the earlier idea is right there to reconsider
• Latent: the steps are hidden states. Models such as Coconut and CODI feed each thought back as the next input, so later steps can reference earlier thoughts, while looped models such as Huginn (Geiping et al.) iterate over the same set of layers on a state that gets overwritten, so earlier content is no longer accessible.

As latent backtracking cannot be identified as its text counterpart, where you just have to look at the output words, we investigate it by passing each step’s hidden state through the output layers, and analyze the situation of the candidate answers.

Backtracking should show up as the top answer changing, as it “drops” the current idea in favour of another one, and this should be reflected in its top choices. We adopt Cui & Ye's interpretation of backtracking, which counts when the top answer changes. Cui & Ye report the winner flips on 32% of Huginn's questions, and it's more likely to get the answer right when it happens.

By establishing that backtracking requires a “change of mind”, to determine whether latent models even backtrack, we wanted to analyze each flip (where the order of the top candidates changed) and see if we could figure out whether it is a revisit of an earlier idea, or an override where the state just keeps on moving until it settles on a different answer, without reading anything from early thoughts.

However, an answer leaderboard change could be caused by multiple factors:

• A real reversal: the top candidate switches, and the model’s final output is affected by it
• Confidence buildup: the same top candidate, but it increasingly distances itself from the other potential outputs
• Noise: two close candidates alternating by a small margin

We can classify only the first one as a genuine change of mind, but before we could investigate the phenomenon, we had to find one.

Searching for the phenomenon in classic model architectures

We started with a simple graph model: Coconut-style, two layers, trained from scratch on graph reachability, with two possible candidates per question (from now on, when we refer to candidates, we refer to a node in this pair). The model achieves 95% accuracy, and we had two separate trained runs, evaluated on the same 419 test graphs.

Throughout the reasoning steps, leaderboard changes occur frequently. The candidate pair’s winner swaps in 29 to 38% of transitions from one reasoning step to another, but a random node pair’s winner flips in 32-42% of the cases as well. On closer look, most last-step flips are the correct answer taking the lead (129/134 and 151/154 for the two trained runs), so this is the final answer settling in, not really a change of mind. Sequences with two or more flips (the natural structure of a true answer reversal) are more frequent for random node pair flips (0.34) than for the candidate pair (0.15). The two training runs only agreed on flips 31% of the time, so they are specific to the training run and not just dependent on the problem.

The readout tracks the search frontier across the graph, so a node’s score gets boosted when the search reaches it. This means that the wrong candidate, if it is reached towards the end, will get a late rise too, but it will not be that close to the top.

​What if the model considers both answers from the candidate pair at the same time, but the readout only shows the winner? After checking how strongly the wrong candidate is represented, we found that it is well above the range unrelated nodes fall in. It distances itself late, and by the time it has reached second place, the winner is already even further ahead. No graph had the two candidates as the top two choices throughout the reasoning. A flip would imply the two candidates were heavily considered at the same time, close together, but this never happens.

​A causal readout (using a Jacobian lens fitted per step to the directions the model's answer actually responds to) paints the same picture, and an untrained model flips less, so we can be more certain that the readout is truly reading the search frontier.

Does latent backtracking happen where it was reported?

Huginn is a 3.5B latent reasoning model with a core block of layers that iterates 32 times over a state, starting from a random initial state and settling over time. Cui & Ye report that mid-reasoning, if the state after each loop is passed through the output layers, the top answer changes on 32% of questions. Additionally, when it happens, accuracy improves, and they claim this is a result of latent backtracking.

There is a disclaimer, though: Cui & Ye evaluated Huginn on 260 questions of their own, with 30 loops and 25 shufflings of the answer order, but they are unreleased. Instead, we used ARC questions, 32 loops, three random initial states, and only one permutation. We reproduced their definition, not their exact experiment from which they made the claim. We ran 50 ARC-Challenge questions and 200 ARC-Easy ones, each one from three random initial states.

​First, we validated our build by matching the 0.38 ARC-Challenge accuracy reported in the Huginn paper (however, on ARC-Easy our prompt reaches 0.5 against their reported 0.7, under a different scoring method). Then, following their definition of backtracking, we found it occurs in 66% of the ARC-Challenge and 64% of ARC-Easy questions, even though they reported 32%. Additionally, their reported improved accuracy on questions where flips happen did not occur: 0.39 versus 0.35 on Challenge, and 0.49 versus 0.48 on Easy.

Are the changes actual decisions the model makes, though? The results say they aren’t. They are caused by the random start settling: the gap between two answers right when they swap has a median of 0.12 logits. If we consider the same question but with a different random initial state, the same gap, read at the same loop iteration, differs between the two runs only by a median of 0.12, so we cannot assert that the changes are more than random alternations. If we rerun the same question with a different random initial state, the flips coincide in only 4% of the cases.

​The right answer swaps with the wrong runner-up no more often than two wrong answers swap with each other in any loop. If we count a swap only when the gap before it and the one right after it are larger than 95% of the noise from the initial state settling, only 1 out of 176 top-answer swaps remain.

The answer to the problem materializes between the 4th and 16th loop: in half the questions, the top answers change in the first 8 loops, in a further tenth of the questions, it happens between the 8th and 16th loops, in 4% between the 16th and 24th and in only 1% it happens after the 24th loop. Looking at the gap between the top two answers, it grows from 0.3 at the 4th loop to around 0.6 at the 16th loop, where it also stabilizes: the leaderboard at the 24th loop differs from when we let it run to 64 loops in only 2 out of 50 problems. This stabilization is not surprising, though, as this is what Huginn was designed to do, while the first loops are the random state settling.

By default, Huginn runs with a compact number format (bfloat16), rounding alone moves answer scores by 0.03 to 0.06, and the gaps we measured were usually around 0.12. However, even after redoing the experiment with full precision (float32), nothing changed: the changes happened at the same frequency, between the same answers, and more than half of the gaps are under 0.13. This means the near ties are not just a rounding discrepancy, but we can’t conclude they are decisions either.

For the first 50 ARC-Easy questions, reading the answer from the 13th loop (before the model had time to settle, right in the middle) instead of the last bumped up the accuracy (0.66 from 0.54). On the full 200-question set, the accuracy improved by only 0.05, and on the 150 questions not included in the first set, the effect is not statistically significant (the interval includes zero), so we treat this as an observation, not as a result.

The backtracking exploration phase Cui & Ye reported is just the random initial state settling.

What does a real, documented change of mind look like?

It is already established that text models visibly reconsider and sometimes change their top answer along the way. Where they were reported, latent flips turned out to be just noise, so could we compare them to a real, text-proven change of mind under the same readout?

​Reasoning models trained with RL started writing tokens such as “wait”, “hmm”, “actually” on their own. These are known to be moments where the models reconsider, so they have become the standard signal for backtracking in text. Steering vectors are trained on them (Venhoff et al.), Thought Anchors (a method that finds which sentences in a reasoning transcript the final answer depends on, and ranks them) identify the “wait” tokens’ corresponding sentences as the ones the later reasoning depends on. At the same time, NoWait shows they can often be deleted without a significant drop in accuracy. If a change of mind actually happens, it’s natural to assume it would be here.

​However, we found it does not signal a change of mind: on a 1.5B distilled reasoning model, we can read the leaderboard by including a short phrase like “the answer is” and seeing each candidate’s score there. We read it in windows covering the “wait” token, alongside matched windows elsewhere in the same transcript. Flips happen less at “wait”, and the leaderboard is more consistent than elsewhere. The model usually commits to the top contender within the first sixth of the reasoning. The five stated reconsiderations we identified in the 24 transcripts we evaluated on actually correct a step (“Wait, that's not right. Heat flows from a hotter object to a cooler one”), not the answer itself. In ReasonOps (a survey of 44,662 transcripts across 12 models), only 1.6% of backtracking events in the reasoning transcripts lead to an answer change, while the rest are plain local repairs.

This should not, however, be overstated as “real reversal is not a thing": it is visible through the readout. From 193,767 R1 math reasoning transcripts, we kept 198 where the model changed the answer it committed to along the way (also, this happens in only 0.1% of transcripts, while in the dataset the backtracking label fires on 42.6%).

If we analyze around the switch, the readout moves by +6.05 logits toward the new answer it commits to, while it moves by +0.29 for a pair of two random answers in the same reasoning transcript. When the answer change is only a rounding of the same number, it is not reflected in the readout, as it shouldn’t (-1.17 for the real pair, -1.13 for the random one). ​An additional interesting case study: if the new answer differs from the old one by a single character, even if it changes a lot in meaning (from 1944 to -1944), it is not visible through the readout, because it scores the whole answer string and the sequence they have in common dominates.

​A change of mind is rare even in text models, and it isn’t where the “wait” is. When it does happen, the readout moves, and far more than for a random pair, which never happened on latent models. This leaves us with the question: what are the latent steps doing instead?

Could we have missed it?

To check whether we can identify a change of mind through the readout at all, we created our own artificial ones. We took a question, ran it through a model, and blended its latent state in varied amounts into the last thought of a different question. At full strength, the model usually gives the answer that was blended in.

When the answer successfully changed, did the readout show a swap? And when it didn’t, did the readout stay unaffected?

• Graph model: the swap showed up on about two-thirds of the questions where the answer moved, and on about a tenth of the others. A natural change of this size would have been spotted, and none appeared.
• Huginn model: the swap showed up regardless of whether the answer changed, on a third of the questions with nothing blended in at all. A third is about the same as the 32% they reported: it’s what this readout gives us when nothing is happening, so on this model it cannot tell a change of mind from noise.

What are the latent steps even doing?

For the latent models we analyzed, the final output is the result of a confidence buildup, not the conclusion of an exploratory phase.

• Graph model: when a leader is distinguishable at a latent step, it is the final output on 96-99% of the graphs, and its lead roughly triples until it reaches the last step.
• Huginn model: the model decides on an answer and becomes increasingly more confident up to a point, and after that, it remains constant. The decision point is not fixed, there is no close runner-up, and more loops do not improve the accuracy (which aligns with Huginn’s designed convergence).

The revisit-versus-override question from the first section implies a reversal. Without one, there is nothing to analyze.

Why is the question hard to ask?

We cannot directly show that backtracking simply does not happen, only that nothing beats noise when measured with an instrument that catches the reversals we can create. Because we found no naturally occurring reversal in latent models, we analyzed artificial ones we generated ourselves, as shown above. We could not find reversals in latent models, and their absence is the finding itself.

Latent backtracking is not easy to investigate because the only known way to catch a glimpse is through the readout, which can vary for reasons outside the decisions the model makes. It is affected by the search frontier expanding, the random initial state settling, and the phrasing of the text sequence we use to identify and score the candidates at that position for text models (rephrasing it from “the answer is” to “so the final answer is” moved the readout by 2.5 logits across three phrasings). A top-answer change only needs two scores to cross, by any amount, and that can happen by chance a lot with four possible answer options and 32 loop iterations. On Huginn, with nothing done to the model, the correct answer swaps with the runner-up in a third of questions. A backtrack that does not change the top answer is invisible to this definition, and to us.

Latent model architectures do not cooperate with the nature of latent backtracking. Revisiting is only possible in feed-back models, but because the answer is bound late at the final step on the feed-back model we used, flipping never happens. Even though reversals were claimed in looped models, by construction there is no way to revisit earlier information, as the state is repeatedly overwritten. The revisit-versus-override question needs both revisitations and reversals, but no model we can evaluate does both. The other feed-back family, CODI, has an open-ended arithmetic task with no prespecified possible answer set, so the comparison with random answers or noise has no obvious structure. The definition is ambiguous and subjective to settle, which is why we excluded CODI from the analysis.

​ The revisit-versus-override question becomes answerable again if a natural latent reversal occurs, whose change is larger than that of the random initial state settling and the alternations of a random pair of answers. RecurTrace lets a looped model refer back to earlier loops, which makes the revisit channel Huginn lacks possible, and it would be the next natural thing to explore.

​All previous results are from evaluating two latent models, each with its own task, and an additional text model for comparison. Everything was run on one 16GB Mac.

GitHub repo: github.com/star2vec/lattrack