I forked Ninfer 3090 and converted it to run on the CMP170HX - doubled my Qwen3.6-35B from llama.cpp

Good afternoon, everyone! I wanted to show the work I've been doing around porting Ninfer over to the CMP170HX ( Github ) So, first, I do want to call out the amazing work that Neroued , Sergiuszm and specifically Don-Chad have all done, to not take away from vLLM, Llama.cpp and SGLang, but to enable local AI to squeeze every ounce of performance we can out of our hardware at home. Free tokens are free tokens and the work that these folks did 100% enabled this fork. I also want to preface early on in this post that I am not a developer, a coder or anything like that. I am an IT Architect so I do know tech and I have been pretty active in the AI community here for the last year. Hermes, powered by Codex and local Qwen3.8-27B, powered by the 4090-fork of Ninfer, were my developers on this project and I just guided them. I have been playing with the 3090 and 4090 Ninfer the last week or so, testing, validating, configuring to work with Llama-swap, and also worked through an NInfer/llama-swap telemetry compatibility gap so llama-swap could capture NInfer’s timing and throughput metrics. BUT getting 30-50% or more performance bumps consistently, for "free", inspired me. Being that the RTX 3090/3090Ti and the CMP170HX were cousins, I ran some recon tests against Don-Chad's repo and my local instance of CMP170HX, unlocked by the CMPUnlocker ( got lucky and got my card <$1000) and was pleasantly surprised that the plumbing was mostly compatible. So Hermes, between GPT-5.6-Terra as the orchestrator/validation and Qwen3.8-27B as worker, found the scaffolding needed to convert. Again, not going to pretend I understand most of this (my background is in VMware, storage, infrastructure stuff. Hermes said: "Getting NInfer working on the CMP 170HX took more than adding sm_80 to the compiler flags. The RTX 3090 fork assumed an 82-SM sm_86 device, while the CMP exposes 70 SMs at sm_80. Several cooperative GDN kernels therefore launched grids that could not be resident all at once, causing cudaErrorCooperativeLaunchTooLarge. I changed the planner to read the actual SM count and dynamically fall back through smaller split-K schedules—16→8→4→2—before using a safe non-cooperative kernel. Workspace sizing was updated to follow the schedule actually selected. I also replaced a separate launch policy hard-coded for the RTX 5090’s 170 SMs, excluded Blackwell-only NVFP4/W4A4 kernels from the SM80 build, and adjusted the SM80 numerical qualification where Ampere reduction behavior differed slightly. In the container, CUDA’s forward-compatibility libcuda had to be removed so the CMP could use the host driver normally. After that, Qwen3.8-27B and Qwen3.6-35B-A3B both loaded and generated successfully with MTP and large KV-cache reservations." The result ended up being a 2x increase in performance on Qwen3.6-35B with up to 262K context configured with PLENTY of headroom (with int8 kv cache and 262k, 26GiB) - see below for llama-swap configuration, which also requires the llama-swap compose configuration that enables calling docker from the host - all this runs on CUDA 13.1.2 runtime / Ubuntu 25.10 "Jarvis": cmd: | docker run --init --rm --no-healthcheck --name ninfer-jarvis-sm80 \ --network container:llama-swap \ --gpus all --ipc host --shm-size 16g \ -e CUDA_SCALE_LAUNCH_QUEUES=4 \ -v /home/your/models:/models:ro \ -v /home/your/llama-swap/logs:/logs \ -v /tmp:/tmp \ cmp170hx-ninfer:sm80-metrics-r1 \ /usr/local/bin/ninfer-serve \ /models/Qwen/qwen3.6_35b-a3b.ninfer \ --host 127.0.0.1 --port ${PORT} \ --model-id qwen3.6-35b-a3b --device 0 \ --max-context 262144 --kv-capacity 262144 \ --max-concurrency 1 --max-pending-requests 4 \ --prefill-chunk 1024 --kv-dtype int8 \ --spec mtp --draft-tokens 3 --lm-head-draft \ --vision --preserve-thinking --no-cuda-graph \ --temperature 0.7 --top-p 0.95 --top-k 40 --min-p 0.0 \ --presence-penalty 1.5 --frequency-penalty 0 cmdStop: "docker stop ninfer-jarvis-sm80" ttl: 300 useModelName: qwen3.6-35b-a3b env: ["CUDA_VISIBLE_DEVICES=0"] My configuration in llama-swap for just in time container loading My specific use case for the CMP170HX and this is for the family's main model that powers Jarvis (replacement for Alexa). The faster I can get everything working at the model level, the faster Home Assistant works, the faster HA Voice works and the sooner I can get everything Amazon ripped out. The screenshots above tell the story of llama.cpp Qwen3.6-35B and Ninfer Qwen36-35B. The story for Qwen3.8-27B isn't as strong being that MoE is memory bandwidth bound and Dense is somewhat compute bound. I've seen, depending on the prompt a 10% bump or a 35% bump in testing, so YMMV. But 2x consistently on both text and image processing on 35B, yes please. llama-swap + CMP170HX ninfea processing Here's a screen shot where PP was over 4000 and TG over 210 on a single request (this was from an Home Assistant API call via HA Voice). I know the CMP170HX is kinda of a hot topic right now and a little more niche than the 3090 and 4090 work but I think this has some real-value. If anyone has issues, or recommendations on how I can make this better, please let me know and I hope someone finds this valuable