[DeepSeek-V4.1-Flash-1017-Genesis] Solving the "Noise Gate": AdamW Heteroscedastic Noise in LLMs

Here is the link: huggingface.co/LuffyTheFox/DeepSeek-V4.1-Flash-1017-Genesis Hello everyone. I have distilled the training noise from the tensor weight spectrum of the DeepSeek V4.1 Flash 1017 model while preserving its learned knowledge, and I am sharing the result with you. This process improved the model's understanding of user-written System Prompts and filtered out noise tokens from the context during conversations. History DeepSeek-V4.1-Flash-1017 is my absolute favorite model. It is smart, blazing fast, and efficient. It solves incredibly complex tasks. However, it has a problem: it writes walls of text to the user, consuming too many tokens. When a user reads these walls of text, they lose track of the core message quickly. Why does this happen? The AdamW Problem The main problem lies in the training mechanism itself, specifically within the AdamW optimizer. Its code doesn't break large values, but it destroys small ones. The problem is hidden right in this line of code: param.addcdiv_(exp_avg, denom, value=-step_size) This line is the heart of AdamW. It contains a mathematical flaw that specifically destroys small singular values. Here is why: 1. Normalization by the square root of the variance. denom is the square root of exp_avg_sq , the second moment of the gradient. For large, stable weights, the gradients are small and predictable. Their exp_avg_sq is small, so denom is small, and the update step ( exp_avg / denom ) remains controlled. AdamW gently nudges such weights without causing chaos. But for small singular values, which are responsible for subtle connections, gradients can be very noisy and have enormous variance. Their exp_avg_sq becomes enormous. In an attempt to "normalize" this noise, AdamW divides by a huge denom . As a result, the update step becomes tiny but chaotic. It does not move the weight toward stability but instead randomly jerks it, creating parasitic singularities. 2. Momentum. exp_avg accumulates momentum. For large weights, this is helpful - it smooths the path to the minimum. But for small, noisy weights, momentum becomes deadly. It causes AdamW to "shoot past" the correct value, creating "spikes" and "bumps" in the loss landscape. This inertia is like a heavy ball that cannot stop precisely near a small target and crashes into it, causing destruction. 3. Weight Decay as a discriminator. Weight Decay ( param.mul_(1 - lr * weight_decay) ) proportionally reduces all weights. For large values, this is a negligible loss. But for small ones, which are already on the verge of disappearing, it becomes fatal. Weight Decay simply "wipes them out," turning them into zeros, and the quantizer then finishes the job. Why don't PyTorch developers change AdamW? The current solution represents an optimal compromise, as the optimizer remains stable with large weights. Furthermore, it is fast and computationally efficient. People don't change the optimizer because it ensures correct convergence during training. AdamW is the ideal tool for training large, stable components. But for fine-grained structures, it is like an elephant in a china shop. Its mechanics - division by noisy variance, inertia, and weight decay - systematically destroy small singular values, creating the "Noise Gate" that I have been trying to fix over the past year. How does this AdamW optimizer problem affect the inference process? During inference, the model relies on matrix multiplications between tensors to generate responses. Since the tail of the tensor’s weight spectrum (small singular values) contains heteroscedastic noise, this increases the error during the inference process. Consequently, the error compounds as the model's context window expands. What did I do? I converted the heteroscedastic noise that AdamW introduced into the tensor weight spectrum into homoscedastic noise. As a result, during inference, the error during matrix operations in the LLM becomes predictable, and the model can keep it within a safe range of values. "Only a fool needs order; a genius masters chaos." Thanks for reading ^_^ PS: Huge thanks to the DeepSeek-V4.1-Flash-1017 model for helping me finish this project. I am deeply impressed by the complex tasks this model can handle. My work is my gift to the whale.