Qwen3.8-Next-Flash up to 240t/s on single rtx 6000 pro

Stumbled around a post about optimizing new Qwen up to 178t/s with a patched version of sglang : github.com/jpezzulli/sglang-rtxpro6000 I managed to reproduce results (kudos to jpezzulli, whomever you are) and spotted a few room for additional speed up (theoretical bandwitch limit for nvfp4 of this model sits at around 280t/s) so I let fable iterate for a few days to push it further. Results: almost 240t/s decode speed on a single rtx 6000 pro maxq (300W). Patch is targeted at huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 , and is applied directly to the main sglang library. Tricks involved: - further quantization of lm head and a few more layers from bf16 to fp8 (reduces bandwitch usage) - kernel tuning of a few layer to improve GPU efficiency - MTP config tuning Additional tricks I'll test over next few days: - further quantization of layers to complete nvfp4 (might not be worth it as it might dégradé performance) - further kernel tuning to fuse some layers - post training of the MTP, and optionally try to create an eagle3 head (the real gain lies here, but not sure how it will faire under real test) Ressources to reproduce: - git: github.com/gabrielolympie/sglang-flashnext-sm120 - repo I started from: github.com/jpezzulli/sglang-rtxpro6000 - model checkpoint: huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 Feel free to propose additionnal tricks to test if you have ideas about how to make it go faster :) Config it was tested on: - Ryzen 9 3950x (16/32 cores) - 128gb ram @3600 (holds the ngram table) - 1x rtx 6000 pro maxq