CUDA: fuse FFN gate/up matmuls and GLU in MMQ with interleaved gate/up rows by anujj · Pull Request #29948 · ggml-org/llama.cpp

time to download some speedup DGX Spark (GB10), llama-bench -ngl 999 -fa on -p 16384 -n 0 -b 2048 -ub 2048 -ctk f16 -ctv f16 -r 5 , tok/s: model type master PR Qwen3.8-27B Q4_K_M 840.1 899.9 +7.1% Q5_K_M 840.1 880.6 +4.8% Q6_K 751.2 785.8 +4.6% Q8_0 876.2 940.1 +7.3% IQ4_XS 882.4 948.0 +7.4% Qwen3.6-27B Q4_K_M 816.6 865.6 +6.0% Q5_K_M 800.0 851.1 +6.4% Q6_K 720.8 770.4 +6.9% Q8_0 876.9 940.2 +7.2% IQ4_XS 882.2 948.5 +7.5% Llama-3.1-8B Q4_K_M 2749.3 3000.6 +9.1% Q5_K_M 2699.4 2960.3 +9.7% Q6_K 2392.2 2616.0 +9.4% IQ4_XS 2933.9 3249.0 +10.7% RTX 5090, llama-bench -ngl 999 -fa on -p 16384 -n 0 -b 2048 -ub 2048 -ctk f16 -ctv f16 -r 5 , tok/s: model type master PR Qwen3.8-27B Q4_K_M 3688.4 3790.9 +2.8% Q5_K_M 3589.5 3659.6 +2.0% Q6_K 3220.5 3281.0 +1.9% Q8_0 3617.3 3766.9 +4.1% IQ4_XS 3796.2 3926.8 +3.4% Qwen3.6-27B Q4_K_M 3567.9 3660.3 +2.6% Q5_K_M 3429.8 3525.3 +2.8% Q6_K 3105.2 3187.8 +2.7% Q8_0 3628.7 3788.4 +4.4% IQ4_XS 3785.6 3915.4 +3.4% Llama-3.1-8B Q4_K_M 12071.5 12625.9 +4.6% Q5_K_M 11823.0 12297.3 +4.0% Q6_K 10513.1 10928.6 +4.0% IQ4_XS 12997.6 13603.3 +4.7%