Fork Around and Find Out Part 3: Interpreting the knight auditor
Summary
This is the third and final post in a series detailing our attempts to mechanistically interpret one head of the chess transformer Maia-3. We find that previous causal evidence gave us a spurious picture of the head, and that rather than just detecting knight forks, head 5 of layer 5 (l5h5) predominantly serves to suppress pointless knight moves.
In this final post, we show that intentionally crafted probes shatter this interpretation of l5h5, and what we once called a “fork detector” is more akin to a “knight move auditor,” one that grades knight moves by their threat level, and is predominantly defined by its strong suppression of poor, quiet knight moves. This has some interesting mirrors to themes in cognitive science research which will be discussed at the end.
Interactive demos for the post can be found here:
The analysis and visualization library:
https://github.com/chessformer-lens/chessformer_lens
Recap
The terminology we have been using:
layer 0–7 — the eight transformer blocks; each contains an attention(sub)layer and an feed forward MLP (sub)layer.
residual stream — the shared communication channel that attention and MLP additively read from and write into; the additivity is what makes per-component decomposition meaningful.
readout point — the residual stream's content at a given depth. We read at 18 points: the input embedding, the post-attention and post-MLP points of each of the 8 blocks, and the final encoder output. These point are after layer norm. Depth labels in figures read [emb, a0, m0, a1, m1, …, a7, m7, enc] (aN = block N's attention, mN = its MLP).
logit — raw output of a machine learning model forward pass before being transformed by an activation function to fit match with the desired predictions
logit lens— decode a readout point through the final unembedding as if it were the finished product, and read the resulting move distribution.
decodable move— a move scoring highly under the logit lens at some depth.
policy — the network's output distribution over all 4352 moves, also used for the output of a single move. Maia plays directly from this; there is no search.
ontology — the model's internal world and how it carves what it knows into categories
Our model:
Maia-3 is a chessformer model built to mimic human play across skill levels, and thus, we hypothesize, internally represents human recognizable features like knight forks. Each square maps to one token. Its attention heads combine two schemes: standard "semantic" QK self-attention, and a "geometric" GAB scheme carrying positional structure. GAB heads are generated per position by a a small generator network.
The running example we have been using is the following position and knight fork Ne6+:
Last post we discovered that knight forks become decodable at layer 5 of Maia-3, and by ablation we found that head 5 is unquestionably the causal link:
We performed an ablation sweep on each head for 300 random knight forks and again find l5h5 is by far the most causally active among all layers' components. We have found this method to be especially useful for this and similar projects.
------------------------------------------------------------------------------------------------------
Part 3
The first domino falls
Although the results so far strongly indicated that we localized a fork computing head and that the model at least partly organizes its ontology by tactics, we opt to run a few more probes, which end up shattering this hypothesis.
Since every forward pass above was performed on a knight move, we now consider the identical tactical motif with a different piece, a pawn fork.
One mined example below:
Upon ablating l5h5, and running N=250 ablations on different such pawn forks, we find that the K+Q fork by a pawn rather than knight has an average logit change of +0.10—the same tactic is essentially unaffected by our head when a different piece is used. The head is not about general forks.
In fact, (N=300) pawn forks yield the full ablation sweep heatmap below
This looks nothing like the sparse knight heatmap with its single dominant carrier head
A gradient of threats
Next we mine the lichess.org database for non forcing knight moves that attack a defended or worthless target, such as a movable rook, and perform the same ablation on N=400 of each class of attacked piece. This is to begin to untangle any structure in the value of the threat the knight creates.
Hollow attacks on pawns are slightly supported, hollow attacks on the more valuable bishops are neural, hollow attacks on the even more valuable rook are suppressed and hollow attacks on the most valuable queen are quite suppressed.
We also look at non-forking knight moves that are strong and non-hollow, for instance in the position below in which Ne5 wins tempo on the queen and centralizes the knight in a hole where it cannot be booted away by any weaker piece.
And again a K+Q fork on a similar board:
So Ablation hurts the logit the most when the knight move has real merit. Consider the difference between a useful queen attack and a hollow one (-1.13 vs +0.56). Head 5 acts on knight moves and scales its support with the value of the genuine threat the knight move actually creates.
The good, the bad, and the pointless
We can also decompose our moves by considering the threateningness of the knight move, mining for "good" vs. "bad" moves, and "quiet" vs. "attacking" moves. We define the predicates by:
Good: the move is the model’s top (rank 0) policy
Bad: the move is legal but the engine ranks it >= bad_min_rank with policy probability <=bad_max_prob (we use 10 and 0.02).
Attacking: the move creates a new attack that genuinely demands a response (target is undefended or is worth more than the attacker).
Quiet: the move creates no new attacks
None: any captures or promotions or checks or blunders (to avoid confounds)
Across all mined good/bad x quiet/attacking knight moves, we find a very interesting foursome of effects.
The head supports good moves and suppresses bad moves, but more strikingly, its largest effect besides game-winning K+Q forks themselves is in the head's suppression of bad+quiet—what we call "pointless"—moves; it targets aimlessness rather than showiness: pointless knight moves, e.g., an unnecessary displacement of a knight from a key central square, are suppressed 2.93 times harder than bad knight attacks. These pointless moves are the sorts that stronger players quickly and intuitively discount for their lack of strategic merit.
The knight auditor
We believe this is enough data to settle the mystery of Maia-3's layer 5, head 5. It is knight exclusive—pawn and other piece moves are untouched by the ablation. It delivers increasing support for genuine threats of increasing value, and a knight forking King+Queen is simply the epitome of this gradient. As the hollow probes showed, if there is no genuine threat we find a gradient of suppression along the value of the attacked piece: P < B < R < Q.
Its most descriptive and distinct role, though, is that it harshly suppresses pointless moves: a knight move auditor.
Concluding remarks
The finding that the head in question works predominantly by suppressing pointless moves mirrors long-standing findings about human expertise from Adriaan de Groot: master chess players are distinguished much less for their deep calculation than for their fast and intuitive suppression of bad moves. Jose Capablance, a world champion renowned for his intuitive understanding of the game famously quipped: "I see only one move ahead, but it is always the correct one." We believe we have found a mechanistic echo (though not necessarily proof of a similar mechanism) in our knight auditor head. The head will strongly suppress the vast majority of bad moves, which are quiet and bad; and to a lesser degree it will suppress the remainder attacking but bad moves, possibly because these usually can not be ruled out as quickly since they can involve violent forcing sequences that must be followed through to some depth.
Appendix
A mechanistic look at layer 5
Remarkably and fittingly, we see that the model learned a sort of "knight pattern" detector for the GAB component of the 5th layer's 5th geometric head. This may or may not be causally relevant to why the attention score boosts the fork logit so much: further research has shown that, for one, one GAB pattern that appears just like a bishop pattern has no relation to where bishop moves become decodable.
For thoroughness, the exact makeup of our knight GAB head is the below linear combination of computed templates for the position:
It is hard to visualize this in one paper. This can be more effectively understood using the chessformer_lens app’s GAB decomposition panel.
Lastly, here is a vivid version of the residual stream and logit lens at each readout point for the running example.
Limitations
- We use the 5 million parameter Maia 3 model only due to its accessibility. The 23m or 79m model may have a different ontology.
- The mining predicates are improvable. They were iteratively written, for instance by realizing that we might be eliding discovered checks and adding a conditional for them, so further improvements may yield cleaner results. This is an especially important area to focus on since the predicates form the basis of our sampling and thus are upstream of every figure and datum.
- We only use one skill setting (high-skill play); lower-skill representations are to be followed up
- The logit lens reads everything in the output basis or vector space with the output's neuron number's dimensions. This means that a feature that is represented in some layer outside the span of the unembedding layer's span will not be detected, constraining our results to a subspace of decodable representations only (LessWrong, 2020).
Acknowledgements
Thank you to Professor Colin Allen for proofing my writing and rhetoric.
This research was conducted independently; views are the author's own.
Please don’t hesitate to push back on any claims or reach out
The figures were generated by my open source python library chessformer_lens
Pip installable and gettable at https://github.com/chessformer-lens/chessformer_lens, DOI: 10.5281/zenodo.21877655, for anyone interested in performing similar work.
点赞
收藏
点踩
分享