543 tok/s single-request Qwen3.6-35B-A3B on one RTX 5090 over a 65K-token decode

An example TL;DR I have open-sourced NInfer , a from-scratch C++/CUDA inference engine currently specialized for two exact Qwen3.6 checkpoints on a single RTX 5090. Both the engine and the converted model artifacts are publicly available: Github : github.com/Neroued/ninfer The main result: Qwen3.6-35B-A3B sustained 542 tok/s while generating a full 65,536 token completion, on a single RTX 5090, single request. My goal was to find out how fast inference can get on a single GPU (in my case RTX 5090), with a fixed model and fixed weights, after deep, end-to-end optimization. To that end, I threw everything I could at it and built the entire pipeline from scratch: custom quantization, weight layout design, per op kernel optimization, kernel fusion, a dedicated LM head draft, and so on. NInfer is not a general inference engine, it's designed just for certain model artifacts. The currently supported models are: Qwen3.6-27B Qwen3.6-35B-A3B Both converted model artifacts are available on Hugging Face. Under NInfer's quantization scheme, the published artifacts are 16.29 GiB (~5.03 bpw) for Qwen3.6-27B and 20.84 GiB (~4.97 bpw) for Qwen3.6-35B-A3B. The Qwen3.6-35B-A3B results: All MTP results below use a draft window of 3 and NInfer’s optimized LM-head draft path. Each result is the mean ± sample standard deviation across five fixed seeds, after one warm-up run. Long-reasoning runs: Completion length Decode speed MTP acceptance 65,536 tokens 542.8 ± 12.5 tok/s 73.0% ~55,171 tokens 572.9 ± 9.1 tok/s 77.7% ~8,675 tokens 634.3 ± 14.2 tok/s 82.7% I also ran a mixed set of code, translation, story, and structured output prompts: Workload Decode speed MTP acceptance Code 576.5 ± 21.7 tok/s 71.0% Translation 559.3 ± 28.1 tok/s 66.6% Story 395.9 ± 30.9 tok/s 37.7% Structured output 661.2 ± 29.5 tok/s 87.2% MTP0 context-length scaling: Prompt length Prefill speed Decode speed 7,680 15,544 tok/s 271.1 tok/s 64,512 10,809 tok/s 242.9 tok/s 130,048 7,828 tok/s 219.4 tok/s 260,096 5,157 tok/s 188.2 tok/s The Qwen3.6-27B results: NInfer also performs strongly on the 27B dense model: Workload Decode speed MTP acceptance Long-reasoning 174.2 ± 3.3 tok/s 79.9% Code 163.9 ± 6.2 tok/s 72.5% Translation 153.6 ± 11.7 tok/s 65.7% Story 110.4 ± 9.2 tok/s 37.9% Structured output 189.1 ± 15.7 tok/s 88.9% Capability scores: I also ran the published artifacts through AIME25, AIME26, and GPQA-Diamond (0-shot, rule scoring, single sample, thinking enabled, MTP=3). Model AIME25 AIME26 GPQA-Diamond Qwen3.6-27B-NInfer 26/30 28/30 172/198 Qwen3.6-35B-A3B-NInfer 27/30 27/30 169/198 Full evaluation configurations are availble in the repository. Capabilities & limitations For both supported models, NInfer handles text, image, and video input, with OpenAI- and Anthropic-compatible HTTP endpoints. It supports limited prefix caching and a range of sampling parameters. With INT8 KV cache enabled on the RTX 5090's 32 GB, both models can reach their full native context length of 262,144 tokens . Known limitations: Only the two listed models are supported. If a stronger, locally-suitable model drops, I'll jump on it immediately. Only RTX 5090 (sm_120a). RTX PRO 6000 should also work, though some kernel tuning may be suboptimal. No continuous batching. (If no new models land soon, I may look into adding it.) I'd genuinely like to see another inference engine match or beat these numbers — similar quantization size, single request, single RTX 5090, Qwen3.6-35B-A3B. Bring it on.