Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090

I mentioned I'd got a second RTX 3090 on a group chat, and a friend said:

I know this is not really your thing... but let me know how quickly it runs Qwen 3.6 35bn MoE. With only 24gb of VRAM you’ll need to use a 4-bit quantized version and you won’t get a massive context window. But it should still be pretty cool.

He's right that it's not really been my thing -- I've been focusing on my own LLMs recently. I decided to dig in a little, and in particular to play withLlama.cpp, which I haven't used for a while. And then things got a tad out of control, and I wound up doing some relatively detailed benchmarking.

The headline results: I downloaded Unsloth's UD-IQ4_NL_XL quantisation of the model fromHugging Face. With that, using the default Arch build of Llama.cpp, which uses Vulkan under the hood:

Using the GPU only, I was able to get the model to generate at just over 120 tokens per second, and it was able to process the prompt at just less than 2,800 tok/s. However, having the whole model on the GPU didn't leave that much space for the context window -- it was constrained to about 50,000 tokens, compared to the model's native context length of 262,144.

Offloading the FFNs for the first 12 of the model's 40 layers to the CPU managed to reclaim enough VRAM to be able to get the full context length; however, with that setup, things were -- unsurprisingly -- slower. I got just over 65 tok/s for generation, and 600 tok/s for the prompt.

Compiling Llama.cpp myself, in order to get the full CUDA version, helped a lot:

With everything on the GPU, I got 140 tok/s for generation and over 3,300 tok/s for the prompt. That was with a context window of 89,600. So everything was better:-)

It was also easier to get to the full context length; that needed just 10 layers' FFNs to be offloaded, and at that point I was getting 89 tok/s for generation, and about 1,100 tok/s for the prompt.

That was a pretty impressive improvement.

But it also showed that the lack of VRAM on…