Sparsity Comes from Score Centering

Why Post-Training Is Sparse · Part 2 of 3. Builds on Part 1, where the expected update is

$G_i=\operatorname{Cov}_{\smu}(A,c_i)$ and the Fisher caps it. Part 3 asks what policy gradient can still reach.

• Output layer, exact: row

$c$’s Fisher is a coin flip,

$h_k^2,\smu(c)(1-\smu(c))$. A row learns only if the sampler hesitates over its token.

• Hidden layers: the Fisher sees only the spread of the likely candidates, fewer directions than there are candidates. That is low dimension; coordinate sparsity is measured, not derived.
• Without centering, a mean shift adds up along the sequence and pushes gradient where the Fisher is near zero. The SFT gradient is exactly this shift.
• Under training–inference mismatch the shift is numerical noise, and centering is what keeps it out.

1. Output layer: a row learns only if the sampler hesitates

In the output layer the Fisher is exactly sparse: row $c$ gets a gradient only if the sampler is unsure about token $c$.

$ z_a=\sum_k W_{a,k}\,h_k\ \Rightarrow\ \kappa_{(c,k),a}=\mathbf 1[a=c]\,h_k,\qquad \bar\kappa^{\mu}_{(c,k)}=\smu(c)\,h_k, $ $ f_{(c,k)}=h_k^2\,\smu(c)\big(1-\smu(c)\big),\qquad \sum_{c,k}f_{(c,k)}=\|h\|^2\Big(1-\sum_c\smu(c)^2\Big), $ $ F^{\mu}_{\text{row }c}\le\max_s\|h(s)\|^2\cdot\mathbb{E}_{\smu}\big[\#\{t:\ y_t=c\}\big]. $

The weight $W_{c,k}$ touches only token $c$’s logit, so for it the next token is a coin flip, $c$ or not $c$. Its Fisher is the coin’s variance times $h_k^2$: zero when the sampler never considers $c$, and zero when it is certain of it. With tea at 0.5, coffee at 0.4 and 0.1 spread over a hundred thousand other tokens, the total is about $1-0.25-0.16=0.59$, and the tea and coffee rows hold $0.25+0.24=0.49$ of it: 83%.

Along a response, $\mathbb{E}[\smu(c\mid s_t)]=\Pr(y_t=c)$, so a row’s Fisher is bounded by how often the sampler emits its token. A task emits a few thousand of the vocabulary’s hundred thousand tokens; the other rows barely move. The total is also the expected squared logit update of Logit Dynamics: averaged over the drawn token, its $1-2P_c+C(P)$ becomes $1-C(P)$, with $C(P)=\sum_aP_a^2$.

The whole output block, and a bound on concentration $ f_W=\big(\operatorname{diag}(\smu)-\smu\,\smu^\top\big)\otimes h\,h^\top . $

The left factor is the covariance of a categorical distribution. Its off-diagonal entries are negative: raising one token crowds out another. Since $\smu(c)(1-\smu(c))\le\smu(c)$, for any set of tokens $S$:

$ \frac{\sum_{c\notin S}\sum_k f_{(c,k)}}{\sum_c\sum_k f_{(c,k)}}\le\frac{\sum_{c\notin S}\smu(c)}{1-\sum_c\smu(c)^2}. $

2. Hidden layers: few directions, not few coordinates

In hidden layers the Fisher sees only the few directions that tell the likely candidates apart.

$ z_a=w_a^\top h,\quad u_i=\frac{\partial h}{\partial\theta_i}\ \Rightarrow\ \kappa_{i,a}=w_a^\top u_i,\qquad \mathbb{E}_{\smu}\big[\kappa_{i,a}^2\big]=u_i^\top\big(C_\mu+\bar w_\mu\bar w_\mu^\top\big)u_i, $ $ f_i(s)=\operatorname{Var}_{a\sim\smu}\big[\kappa_{i,a}\big]=u_i^\top C_\mu\,u_i, $ $ C_\mu=\sum_a\smu(a)\,(w_a-\bar w_\mu)(w_a-\bar w_\mu)^\top,\qquad \operatorname{rank}C_\mu\le n-1. $

Centering deletes one direction, the mean output vector $\bar w_\mu$, along which every likely logit rises together. What remains is the covariance of the candidates. If the sampler puts mass on $n$ tokens, their deviations from the mean sum to zero, so they span at most $n-1$ directions. In the extreme where $\smu$ sits on one token, every parameter’s Fisher is zero after centering, while the uncentered second moment is not. Centering is what turns a concentrated distribution into a small Fisher.

This is low dimension in function space, not coordinate sparsity. That would need most $u_i$ to be nearly orthogonal to these directions, which depends on the architecture and on pretraining, and the directions change from prefix to prefix. The evidence is empirical. In RL on Qwen3-1.7B-Base, Zhang, Mukherjee & Peng (2026) find that training only the top 0.1% of parameters by initial Fisher recovers 85.21% of the full-training gain on MATH-500, and 5% recovers 98.82%. The token Fisher under the model’s own probabilities is 0.16 of its value under a uniform distribution at the start, and 0.04 after 1,000 steps.

3. Without centering: the mean shift

Without centering, a mean shift adds up along the sequence and pushes gradient into directions the Fisher cannot see.

$ m_i(s)=\bar\kappa^{\mu}_i(s)-\bar\kappa^{p}_i(s),\qquad \mathbb{E}_{a\sim\smu}\big[(\kappa_{i,a}-\bar\kappa^{p}_i)^2\big]=\operatorname{Var}_{a\sim\smu}\big[\kappa_{i,a}\big]+m_i^2, $ $ \mathbb{E}_{\smu}\big[\mathrm{score}_i^{(u)}\,\mathrm{score}_i^{(t)}\big]=\mathbb{E}_{\smu}\big[\mathrm{score}_i^{(u)}\,m_i(s_t)\big]\ne0\quad(u\lt t). $

Draw tokens from $\smu$ but subtract only the model’s mean, as the plain softmax score does, and each position averages to $m_i$ instead of zero. When the shift keeps its sign, the positions add up coherently: roughly $T\sigma^2+T^2m^2$ against $T\sigma^2$ with centering, an order-of-magnitude estimate rather than a theorem.

SFT is the canonical case. There $\smu$ is the data’s next-token distribution, prefixes come from the data, and by the tower property the gradient is exactly the accumulated shift:

$ \frac{\partial}{\partial\theta_i}\,\mathbb{E}_{\text{data}}\big[\log\pi_\theta(y\mid x)\big]=\mathbb{E}_{\text{data}}\Big[\sum_t m_i(s_t)\Big]. $

This is the term centering removes: with a constant signal $A\equiv1$, the centered update is $\operatorname{Cov}_{\smu}(1,c_i)=0$. In the output layer the demonstrated token’s row receives $(1-\pmodel(y^\star))\,h$, nearly full size when the model gives $y^\star$ almost no probability, while the on-policy Fisher of that row is proportional to $\pmodel(y^\star)$. Data that leaves the model’s distribution touches parameters the model itself never touches.

The same cost, booked as a χ² signal

Let $D$ be the data distribution and $w=D/\pi_\theta$. Because the score has zero mean under $\pi_\theta$, any constant $b$ can be subtracted:

$ G^{\text{SFT}}_i=\mathbb{E}_{\pi}\big[w\cdot\mathrm{score}_i\big]=\mathbb{E}_{\pi}\big[(w-b)\,\mathrm{score}_i\big],\qquad |G^{\text{SFT}}_i|\le\sqrt{\mathbb{E}_{\pi}\big[(w-b)^2\big]}\,\sqrt{F_i}, $ $ \mathbb{E}_{\pi}\big[(w-b)^2\big]=\chi^2(D\,\|\,\pi_\theta)+(1-b)^2\ \ge\ \chi^2(D\,\|\,\pi_\theta). $

The weight is a product $w=\prod_t\rho_t$ with $\rho_t=q(y_t\mid s_t)/\pmodel(y_t\mid s_t)$, where $q$ is the data’s next-token distribution, and $1+\chi^2(D\|\pi_\theta)=\mathbb{E}_D[w]$. Conditioning on the last position gives $\mathbb{E}_D[\rho_T\mid s_T]=1+\chi^2\big(q(\cdot\mid s_T)\,\|\,\pmodel(\cdot\mid s_T)\big)$. If every position has $\chi^2(q\|p)\ge c$, backward induction over positions, with every ratio positive, gives

$ 1+\chi^2(D\,\|\,\pi_\theta)\ \ge\ (1+c)^T . $

The bound still holds, but its signal scale grows at least exponentially with distance from the model. It is the same $(1+\chi^2)^T$ form as the variance of sequence-level importance sampling in rl-collapse Part 2, and the $\chi^2$ that makes per-token importance sampling noisy in Around the Sampler. Booked directly under the data, the cost moves into the empirical Fisher $\mathbb{E}_D[\mathrm{score}_i^2]$, which a small $F_i$ does not bound.

4. Four cases side by side

The bound is a property of the estimator, not of RL.

TD weights a raw logit gradient, $\mathbb{E}[\delta\cdot\kappa_{i,a}]$, which keeps the common component: for a Q-value a uniform shift is the value level, not a redundancy. So the bound does not apply, which is not evidence that TD updates are dense. Two observations fit the table: DPO on near-policy preference data is as sparse as RL (Mukherjee et al., 2025), and SFT on expert traces made near-policy by MCMC sampling rivals RL (Karan, Chen & Du, 2026). Both keep $\chi^2(D\,\|\,\pi_\theta)$ small.

5. Sparsity is not a defect

Under training–inference mismatch, centering is what keeps numerical noise out of the update.

When the sampler is an inference engine, $\smu\approx\pmodel$, and the shift $m_i$ comes only from numerical differences between two implementations. It carries no reward information, yet it injects gradient into directions with near-zero on-policy Fisher and accumulates along the sequence. In the output layer the shift for row $c$ is $(\smu(c)-\pmodel(c))\,h$. A token the model rarely samples has a near-zero Fisher, yet when drawn it still receives $(1-\pmodel(c))\,h$. This is consistent with the mismatch doing most of its damage on rare tokens (When Speed Kills Stability).

Centering removes the shift exactly and keeps the update in the directions spanned by $C_\mu\approx C_p$. Trust Region Masking never clips a token, so it leaves this per-token structure intact; it only drops whole sequences that left the trust region. Sparsity here means not moving the parameters that only noise would push.

6. Scope and open questions

• What centering needs. The sampler’s full next-token distribution at every prefix: engine log-probabilities or a stale model, not human demonstrations. With top-k or top-p, it is the truncated, renormalized distribution. Estimating $\bar\kappa^{\mu}$ from a batch makes the cancellation approximate.
• What it changes. Off-policy, centering drops the pull toward the sampler; the coefficient $\beta$ in Around the Sampler adds it back on purpose.
• Expected versus accumulated updates. All of this is about the expected update. Adam normalizes per coordinate, so a small but consistently signed gradient still moves a parameter by about one learning rate; the optimizer and precision decide the rest (step sizes: Keeping the Off-Policy Gap Fixed).
• Two mechanisms, not yet separated. A concentrated sampler and candidates that react alike both lower the Fisher. Two clean tests: permute which tokens get which probabilities at fixed entropy, and measure per coordinate the share centering removes, $\rho_i=(\bar\kappa^{p}_i)^2/\sum_a\pmodel(a)\,\kappa_{i,a}^2$. If low-Fisher coordinates have $\rho_i\approx1$, the sparsity comes mostly from the common component, as an untested heuristic suggests: likely tokens have large logits, so $h$ may lean along $\bar w_\mu$.
• One step. The Fisher itself changes during training, which is where Part 3 starts.

Next. Part 3 turns the bound around: how far can one update move a response the model almost never produces?

References

• Y. Zhang, S. Mukherjee, H. Peng. Understanding LLM Parameter Update Sparsity through the Lens of Fisher. arXiv, 2026.
• S. Mukherjee, L. Yuan, D. Hakkani-Tür, H. Peng. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. NeurIPS 2025.
• A. Karan, S. Chen, Y. Du. Finetuning with Sampling: SFT Learns Better Than You Think. arXiv, 2026.
• Y. Li. Logit Dynamics in Softmax Policy Gradient Methods. arXiv, 2025.
• J. Liu, Y. Li, et al. When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch. Blog, 2025.
• Y. Li, J. Liu, et al. Trust Region Masking for Long-Horizon LLM Reinforcement Learning. ICML 2026.