Running DeepSeek V4 Flash Q4_K_XL at ~100 tok/s prompt processing on 4× RTX 3060 12GB

I managed to run the 143–144 GiB DeepSeek-V4-Flash-0731 UD-Q4_K_XL GGUF on four RTX 3060 12GB cards while keeping a 360k–376k context window. Hardware: CPU: Intel Core i9-10920X, 12C/24T RAM: 128 GB DDR4-3200, quad-channel GPU: 4× NVIDIA RTX 3060 12GB Total VRAM: 48 GB Storage: NVMe SSD Engine: llama.cpp, build b10181 Model: unsloth/DeepSeek-V4-Flash-0731-GGUF Quant: UD-Q4_K_XL, approximately 144 GiB KV cache: Q8_0 The best high-speed configuration so far: llama-server \ -m DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00001-of-00005.gguf \ -c 368640 \ -ncmoe 34 \ -ts 100,1,1,1 \ -ot 'blk.(3[4-6]).ffn_.exps=CUDA1,blk.(3[7-9]).ffn.exps=CUDA2,blk.(4[0-2]).ffn.*_exps=CUDA3' \ -ctk q8_0 \ -ctv q8_0 \ -b 2048 \ -ub 2048 \ -np 1 \ -lm none \ --threads 20 \ --flash-attn on Measured with a roughly 20.5k-token prompt: Configured context: 368,640 tokens Prompt processing: 99.4 tok/s Text generation: 10.1 tok/s Minimum free VRAM under load: GPU0: 671 MiB GPU1: 842 MiB GPU2: 1395 MiB GPU3: 1395 MiB Model load time: approximately 198 seconds Other measured context/safety options: Context Prefill Decode Minimum free VRAM 376832 99.5 t/s 10.4 t/s 611 MiB 368640 99.4 t/s 10.1 t/s 671 MiB 360448 99.4 t/s 10.1 t/s 735 MiB The interesting part is the GPU layout. -ncmoe 34 keeps the experts from blocks 0–33 in system RAM. The remaining nine expert layers are explicitly distributed across GPUs 1–3, three layers per GPU. The extreme -ts 100,1,1,1 split does not distribute those explicitly assigned expert weights. Instead, it pushes most non-expert tensors—attention, KV-related allocations, etc.—onto GPU0. That leaves enough space on GPUs 1–3 for the large expert layers. This was much better than trying to calculate the layout analytically. With -ncmoe and explicit -ot overrides, tensor placement is discrete and somewhat unintuitive, so I measured every candidate. Microbatch size was the biggest performance lever: -ub 1024: approximately 63.4 tok/s prompt processing -ub 2048: approximately 99.4 tok/s prompt processing Decode remained almost unchanged at approximately 10.1–10.5 tok/s. At the full 393,216-token context, -ub 2048 also worked, but GPU0 had only 493 MiB free under load. Reducing the configured context to 368,640 restored a 671 MiB margin without reducing prompt-processing speed. For comparison, the safer -ub 1024 configuration can run with a configured context of 524,288 and still showed about 1032 MiB free on the tightest GPU, but prompt processing drops to approximately 63.4 tok/s. A few additional findings: Q8_0 KV is the default choice. F16 KV at c=393216 left only 587 MiB free. -ncmoe 33 caused a CUDA allocation failure. Memory mapping was disabled with -lm none. -np 1 is important; multiple slots multiply KV-cache requirements. The model is mostly in system RAM, so quad-channel memory bandwidth matters heavily. Even so, getting approximately 100 tok/s prompt ingestion and 10 tok/s generation from a 144 GiB MoE model on four consumer 12GB GPUs is much better than I expected. The configuration has been tested under real prompt load. The entire 368k context window has not yet been filled end-to-end, so the number above is the configured capacity, not a claim that I already completed a 368k-token generation test. Generated by ChatGPT 😂.