Pruning Qwen3.6-35B-A3B for RTX 5090: what I learned pushing MoE compression to its limit on a single GPU

Why this exists

I had a 35B-parameter MoE model that needed to run on a single RTX 5090. The model in FP8 needed 34.4 GiB. The GPU had 31.84 GiB. The gap was 2.6 GiB — small enough to feel tantalizing, large enough to break every standard deployment pipeline.

Six days later, I had a pruned model that scored 73.2% on HumanEval+, 51.0% on Toolcall, and 33.6% on MMLU. That model (v3) shipped as the best result available at the time.

Twenty-four hours after that, I had evidence that everything I thought I understood about calibration was specific to one recipe and didn’t generalize. Then another twelve hours produced v7b-fp8 — a model that beats v3 on every measured pack I ran, with BugFind +17, DataExtract +17, and InstructFollow +20.

The path between those states was not linear. Some of my most confident conclusions turned out to be wrong. This article is what I wish I had known on day one — updated after the sessions that overturned my earlier priors.

Qwen3.6-35B-A3B is a 256-expert MoE model optimized for agentic coding, and my only available hardware was a single RTX 5090. This covers pruning, quantization, evaluation, and attempted recovery fine-tuning on a single consumer GPU across seven experimental sessions. It does not cover multi-GPU setups, cloud inference, or comparisons with other pruning algorithms.

Why a 2.6 GiB deficit ate my week

The standard view is that compressing a model by 8% to fit a memory budget is a routine calibration exercise — run AWQ or GPTQ, adjust the quantization config, ship it. A 2.6 GiB gap on a 34.4 GiB model is 7.5% compression. Well within what quantization alone should handle.

It was not routine.

The contradiction hit immediately: BitsAndBytes, GPTQ, and AWQ all quantize only 2D nn.Linear weights. MoE models store experts as batched 3D tensors — [n_experts, in_dim, out_dim] — for efficient grouped matrix multiplication. These 3D tensors contain roughly 90% of the model’s total parameters, and every standard quantization tool simply skips them. No error. No warning. They just pass through at full precision.

So quantization alone could not close the gap. I needed expert pruning.

REAP (Router-Weighted Expert Activation Pruning) scores each expert by the conditional average of its gating weight times its activation norm over calibration tokens:

$S_j = \mathbb{E}_{x \in X_j}[g_j(x) \cdot |f_j(x)|]$

The intuition: an expert that gets low gating weight and produces small activations contributes little to the output and can be removed with minimal reconstruction error. REAP then removes the lowest-scoring experts and propagates residuals to keep the functional manifold topology intact.

The pruner works block-by-block: load one 750 MiB decoder layer to GPU, score all surviving experts, prune the lowest, propagate residuals, save. About 25 minutes per experiment across 40 decoder layers. I ran seven experiments over seven sessions on one GPU.

The constraint that shaped everything: the RTX 5090’s 32 GiB was simultaneously my inference platform, evaluation framework, and training environment. Pruning, eval, and SFT all competed for the same VRAM, and no two of them could run at the same time. At BF16, the model occupied 30.6 GiB — 97% of GPU capacity, leaving zero room for gradients or activations during training.

This total-resource competition, not the pruning algorithm, turned out to be the hard problem.

The calibration mix that rewrote my priors

I came to this expecting the pruning algorithm or the compression ratio to dominate quality. That is how the literature frames it: better importance scores, better pruning decisions.

The evidence says otherwise.

Here is what happened.

My first pruning experiment (v2) used four code-focused calibration datasets: evol-codealpaca, BigCodeBench, SWE-bench, and xlam. Pure code data. The result: HumanEval+ at 72.0%, Toolcall at 44.0%, and MMLU at — two categories at 0.0%.

Dead categories. The model could generate code, but it could not answer a general-knowledge question. The pruner had never seen general-knowledge tokens during scoring, so it had no way to know which experts mattered for those domains. Any expert carrying general-knowledge information was pruned or severely weakened.

For the v3 experiment, I switched to a 70/30 code-to-general mix. Same REAP algorithm. Same compression ratio. Just two general datasets added — 600 samples of MMLU and 600 samples of C4 — to a pool of 700 samples each from four code datasets.

Category

v2 (pure code cal)

v3 (70/30 cal)

Change

MMLU Social Sciences

0.0%

33.3%

• 33.3pp

MMLU Other

0.0%

34.3%

• 34.3pp

HumanEval+

72.0%

73.2%

• 1.2pp

Toolcall

44.0%

51.0%

• 7.0pp

Every single metric improved. MMLU recovered from dead to 33%+. Code benchmarks improved too.

I think of calibration data as the lens through which the pruner sees the model. A narrow lens (pure code) gives a sharp but myopic view — the pruner keeps only what it sees, and blinds the model to everything else. A wider lens (70/30 mix) lets the pruner see the full functional space, so it preserves the structures that serve the whole distribution, not just one mode.

The analogy breaks in one direction: you cannot just keep adding calibration domains forever. More data means longer scoring passes. But within a practical budget, the evidence is clear — calibration composition matters more than any algorithmic tweak I could have made to the pruner itself.

I shipped v3 and moved on. That was where the story got interesting.

The discovery that my conclusions were calibration-specific

After shipping v3, I went back to test a hypothesis that seemed obvious: if I replaced the code-heavy calibration with agentic traces, the pruned model would perform better on agentic benchmarks. Tool calling, bug finding, multi-step reasoning — these were what the model was designed for.

I built a proper BenchLocal evaluation harness — 8 packs covering tool-call, hermes-agent, bug-find, data-extract, instruct-follow, reason-math, struct-output, and cli — and established a v3 baseline with the new v19 chat template. The baseline was sobering: ToolCall-15 at 90, HermesAgent-20 at 16, BugFind-15 at 8. The agentic gates were low to begin with.

I ran two candidate experiments at a deeper compression (0.40, keeping 154 of 256 experts) with two calibration strategies:

Mix-A: full replacement — agentic data (glm47-reap + hermes-agent-traces) instead of code

Mix-B: additive — agentic data layered on top of v3’s base

Both produced the same result. ToolCall-15 dropped from 97 (the v3 baseline with the old reasoning parser) to 90. HermesAgent-20 stayed at exactly 16. BugFind-15 hovered in the 3-10 range. No agentic movement. Both failed all three gates.

I stopped and ran a control experiment. I took Mix-A’s exact calibration list and ran it at v3’s exact compression ratio (0.289, 183 experts). If compression depth was the cause of the toolcall regression, the control would recover toolcall. If calibration content was the cause, the control would still show the regression.

The control was decisive:

Candidate

Compression

Calibration

ToolCall-15

HermesAgent-20

BugFind-15

v3

0.289

70/30 balanced

97

16

8

Mix-A

0.40

Agentic replacement

90

16

10

Mix-B

0.40

Additive layering

90

16

3

v3ratio

0.289

Mix-A calibration

90

16

0

ToolCall-15 at 90 across all three candidates, including the control at v3’s compression. The regression came from the calibration content, not the compression depth.

The mechanism was surprisingly specific. The regression was concentrated entirely in a single sub-dimension: Parameter Precision dropped from 100 to 67. The model still picked the right tools with the right structure — it just generated wrong-typed or wrong-formatted arguments more often. Dropping code corpora (evol-codealpaca, bigcodebench, swe-bench) and substituting agentic traces had cost the model its tight argument-formatting discipline.

The other finding was…