I didn't know it was IMPOSSIBLE!... I just needed it." - 400k+ Context on Qwen 3.6 MoE 35B on a Single 16GB GPU (RTX 5060 Ti) at 25 t/s

Hi everyone!

People often tell me that something is “impossible”… but when it actually works, they suddenly have nothing to say.
They claim that a 10M context with 90%+ coherence is undoable? Well, Grok publicly rated my solution on X (Twitter) as a 9.8 out of 10… for achieving a 60M context with 90%+ coherence.

Today, I want to share a successful stress-test optimization that allowed me to push the Qwen 3.6 MoE (35B-A3B) model far beyond its native limits on a single consumer-grade RTX 5060 Ti 16GB GPU, while keeping all 99 layers completely in VRAM.

My Setup & Metrics

• GPU: RTX 5060 Ti 16GB (GDDR7)
• Model: Qwen3.6-35B-A3B-12Gb-2.6763bpw.gguf
• Context processed: 353,993 tokens (real usage inside a live coding session)
• VRAM allocation: 15166 MiB / 16311 MiB (with around 1.1GB still free — and this includes the OS + 2 open browsers!)
• Prefill time: 8m 14s (~716 tok/s at the very end of the 353k chunk)
• Decode speed: 24.9 - 25.9 tok/s (Prompt Caching works perfectly on follow-up turns!)

Key Takeaways

1. You can easily push the limit up to 450k tokens without crashing the system. Pushing to 500k is possible, but it’s right on the edge.
2. The model remains completely coherent (I haven’t strictly verified the code execution accuracy at this scale yet, but the logic holds up).
3. Thanks to Prompt Caching, it works incredibly well. Getting 25 tok/s at this context scale speaks for itself. Massive kudos to the llama.cpp team!
4. The smart quant model squeezed down to exactly 11.7GB is beyond praise. Low bow to the author of this quant!

Deep Dive & The Solution (How to bypass HTTP 400 & Capping)

Standard parameters like qwen2.context_length or bare context_length overrides were ignored by the recent llama-server builds with CUDA 13.3, forcing a 256k slot cap. The hard limit of the model was successfully forced-overridden via the correct metadata key: --override-kv qwen35moe.context_length=int:400000.

Here is the exact launch command that made this rock-solid stable:

llama-server.exe --model Qwen3.6-35B-A3B-12Gb-2.6763bpw.gguf --ctx-size 400000 --temp 0.55 --cache-type-k q4_0 --cache-type-v q4_0 --gpu-layers 99 --batch-size 256 --ubatch-size 256 --parallel 1 --min-p 0.05 --repeat-penalty 1.1 --repeat-last-n 64 --n-predict -1 --flash-attn on --override-kv qwen35moe.context_length=int:400000 --rope-scaling yarn --rope-scale 1.526 --yarn-orig-ctx 262144

What do you think? Who can replicate or beat this result?