Deep|LLM: Kimi K3's KV Cache Is Smaller - Why That May Actually Be Positive for DRAM/NAND

Back in late April, we highlighted the launch of DeepSeek V4 as the inflection point for large-scale NAND-based KV cache. We detailed how DS V4 achieved massive efficiency gains in inference by offloading the KV cache to NAND, and expected more frontier labs to adopt a similar methodology soon. Fast forwarding about three months, almost everyone is adopting similar approaches, from Anthropic and OpenAI to Kimi K3, the first Chinese model able to compete at the highest level in coding. By comparison, DeepSeek was able to compress the KV cache by 90%, while Kimi reached around 75%.

Over the weekend, a peer firm’s Twitter commentary on Kimi K3 and its implications for memory demand set off a wave of discussion. We agree with the direction of their core call: Kimi K3 is not a lightweight model that “meaningfully reduces memory and networking demand,” but rather a very large MoE with more efficient attention memory, heavier model weights, and wider Expert Parallelism. K3’s total parameter count reaches 2.8T; it activates 16 of 896 experts per token and uses KDA, Gated MLA, Stable LatentMoE, and MXFP4 weights. Moonshot officially recommends deploying it on a high-bandwidth supernode with at least 64 accelerators. It therefore still needs plenty of HBM, scale-up networking, and GPUs. In terms of performance, it can be considered firmly in the top tier. However, Moonshot itself admits overall capability and user experience still trail the strongest closed-source models, so the hype should be kept in check. Meanwhile, as our earlier report noted, K3’s token efficiency is relatively low, making it more expensive than GPT-5.6 Sol on real long-horizon tasks, with no clear cost-performance edge.

That said, we think parts of the peer’s argument need more precision. First, WideEP does not shuttle the full model weights between GPUs for every generated token; expert weights are typically sharded and resident on each GPU, while the network mainly carries the token activations routed to different experts, with a combine step after compute. K3’s 16 activated experts out of 896 total do meaningfully increase dispatch/combine all-to-all traffic, which is why it favors a large scale-up domain like NVL72 - but the binding constraint is activation routing bandwidth, not repeatedly moving weights. Second, on a rough MXFP4 calculation, 2.8T weights come to about 1.4TB, and exceeding 1.5TB after non-FP4 parameters, scales, and metadata is reasonable. Still, a GB200 NVL72 has roughly 13.4TB of HBM in total, so “model weights alone nearly fill HBM” does not generally hold. What actually drives KV offloading is long context, high concurrency, long-lived prefix caches, and the push for larger batch sizes.

1. What is KV cache offloading, and why it mattersSubscribe nowRead more