NVFP4 on VOLTA! Despite being built for Blackwell, I made four 2017 V100s run Qwen 3.8 NVFP4 natively and match my $6000 RTX 5090.

Four Tesla V100s from 2017 matched my RTX 5090 on single-request Qwen 3.8 decode.

Repo: https://github.com/dnv2003/v100-skinny

https://i.redd.it/5ws2ak3uqckh1.gif

The 5090 was not being held back. It ran NInfer, a specialist engine built to make this exact model as fast as possible on that GPU. (love this guys work)

The V100s ran Qwen3.8's published mixed FP4/FP8 weights unchanged.

This should be impossible. NVFP4 was built for Blackwell. The RTX 5090 has native silicon for FP4 and FP8; V100 has none of these advantages.

And yet via software I wrote a translator fast enough to reach parity in decode.

Here are the same-lab results:

Both sides used temperature 0.6, top-p 0.95, top-k 20, presence penalty 1.0, thinking enabled, and the same five seeds.

The V100 system is 2% ahead in the decode-throughput point estimate. NInfer is about 5% ahead in decode-only time to the correct answer. The intervals overlap.

The honest conclusion is parity.

And this is not a DFlash/EAGLE/n-gram/separate-drafter result. Both systems use Qwen3.8's own built-in MTP, each at its best measured depth on this workload. NInfer is at its maximum supported depth of five; v100-skinny runs at seven(thanks to QPN).

The interesting part is why parity happens.

NInfer turns a round in 19.9 ms. The V100s need 26.9 ms — 35% longer.

But the V100 system commits 5.89 tokens per round against 4.27 — 38% more.

So the slower round and the deeper round almost exactly cancel:

1.38 / 1.35 ≈ 1.02.

NInfer wins each round. v100-skinny gets more useful work out of each round.

That deeper verification only pays because of QPN, the kernel I wrote.

What I actually built

The V100 has no FP4 Tensor Core instruction and no FP8 Tensor Core instruction.

QPN keeps the model compressed while it is read from HBM, then translates each tiny fragment directly into the FP16 register format Volta's existing Tensor Cores can consume.

There is no giant "dequantize the model to FP16 first" step.

At the actual Qwen3.8 per-rank shapes, measured against an 879 GB/s read-only ceiling on these cards:

The important row for the 5090 comparison is M=8.

Volta's tensor instruction naturally works on an eight-row tile. v100-skinny maps a k=7 speculative verification round onto exactly those eight rows, so checking more candidate tokens is unusually cheap.

That is the trick: I cannot give Volta Blackwell's FP4 hardware, but I can restructure the problem around the hardware Volta actually has.

v1.0 got us here. v1.1 removes its last compromise.

In v1.0 I solved the unsupported-FP8 problem by converting those regions into NVFP4, because Volta had no execution path for them.

That made modern NVFP4 serving practical on V100, but it meant serving a derivative checkpoint.

v1.1 gives those FP8 regions a real SM70 execution path too. The model's published allocation can now stay intact:

published FP4 regions stay FP4 → QPN2 published FP8 regions stay FP8 → QPN8 activations → FP16 KV cache → FP16 

So instead of changing the checkpoint to fit Volta, the execution engine now adapts to the checkpoint.

Why preserving the model matters

My earlier all-FP4 Qwen3.8 path could look spectacular under speculative decoding for the wrong reason: damaging the model made some outputs more repetitive, and repetitive output is extremely easy to predict.

On one 50-item hardware-generation test:

Fast nonsense is still nonsense.

That is why v1.1 running the published mixed allocation matters more to me than another synthetic tok/s record.

This is a server, not a GEMM screenshot

The headline result includes the actual 27B model, four-GPU tensor parallelism, attention, recurrent state, native MTP, CUDA Graphs, sampling and an OpenAI-compatible endpoint.

The work also turned up several completely separate SM70 traps:

• the checkpoint's FP8-KV directive sent Volta onto a slow scalar attention path, so production uses FP16 KV;
• the SM70 drafter default was sampling its own proposals instead of using greedy/local-argmax proposals;
• the target verify path had unnecessary state synchronizations and copies;
• declared max context was contaminating decode partition geometry.

None of those show up in a GEMM benchmark. They matter once you try to make the whole model fast.

What about long context?

I also found the point where fixed k=7 stops being the right choice.

At roughly 65K live context:

So the lesson is not "turn speculation off at long context." It is that the best depth changes with context.

At ~65K, each extra drafter step has to traverse the long KV history, while k=7 accepts barely more tokens than k=3. Shallower native MTP still wins.

Automatic per-request depth selection is follow-up work; for now the measured long-context recommendation is k=3 rather than k=7.

Separately, merely declaring a large context window no longer taxes short requests: with the partition fix, round latency is flat to within about 0.25 ms from --max-model-len 4096 through 262144 on the measured short-context cells.

The full 262K window is memory-marginal on my box; 244,608 tokens is the largest configuration that boots reliably across both observed memory profiles.

The obvious caveats

Four GPUs versus one?
Yes. This is a capability/acquisition-cost result, not a density victory.

A$600 computer?
No. My four V100 cards cost roughly A$600 total in accelerator hardware. The server, CPUs, RAM, cooling and electricity are additional.

Power efficient?
Absolutely not. These are 300 W datacentre cards. A 5090 is the vastly nicer machine to own.

Does V100 beat the 5090 everywhere?
No. NInfer's prefill is roughly 4× faster probably more. This result is about single-request decode, where weight bandwidth dominates and the old cards can still fight.

Same quantized checkpoint on both machines?
No. Same Qwen3.8 base model, but this is a best-system-vs-best-system comparison: v100-skinny serves RadixArk's published mixed checkpoint; the NInfer artifact is Unsloth-derived. I am not presenting it as a same-weight causal engine A/B.

Cherry-picked speculative depth?
Each engine is shown at its own best measured native-MTP depth for this workload, and the repo contains the depth controls and raw outputs.

Why I care

You can now run a 27B modern mixed FP4/FP8 model at roughly 220 tok/s single-request decode on about A$600 of retired V100 accelerator cards.

That does not make V100 a better product than a 5090.

It means a lot of hardware written off as "too old for modern AI" is missing less silicon than it is missing software.

The 5090 gets NVFP4 support from the quantization format all the way down to native Blackwell silicon.

The V100 gets none of that.

v100-skinny supplies the missing execution architecture in software.

Repo / quick start / kernels / raw results:

https://github.com/dnv2003/v100-skinny

If anyone still has a C4130, DGX-1 or another four-V100 box around, I would especially like independent reproductions.

Prepared first comment

Methodology / receipts before the recurring questions arrive:

• Repo: https://github.com/dnv2003/v100-skinny
• Reproduction: docs/REPRODUCE.md
• Same-lab 5090/V100 result: results/headtohead_5090_20260819.md
• AIME + seconds-to-answer: results/aime_partfix_20260819.md
• Kernel matched benchmark: results/kernel_matched_20260819.csv
• Long-context/depth sweep: results/ctx_depth_20260819.md
• Native mixed-path regression: results/mixed_regression_closed_20260818.md

A few specifics:

• 4× V100-SXM2-16GB vs 1× RTX 5090.
• ~A$600 is what I paid for the four GPU cards, not the complete server.
• Both sides are server-side decode measurements, not UI/rendering speed.
• Both use Qwen3.8's native MTP. No DFlash, EAGLE, n-gram speculation or separate draft model.
• V100 headline depth: k=7. NInfer: draft-tokens=5, its best measured and maximum supported depth here.
• Sampling is matched: temp 0.6 / top-p 0.95 / top-k 20 / presence penalty 1.0 / thinking on.
• Both went 5/5 on AIME 2026 problem 1 across the five fixed seeds.
• At ~65K live context, k=3 is currently the right V100 profile: 76.3 tok/s vs 65.5 with MTP off and 54.7 at k=7.
• Prefill is not parity: NInfer is roughly 4× faster there.
• The head-to-head is same base model / different published quantized artifacts, and is therefore a system comparison rather than a same-weight engine ablation.
• The four V100 cards are loud, power-hungry 2017 datacentre hardware. That is part of the point, not something I am hiding.

Upstream credit: v100-skinny builds on 1Cat-vLLM, which made modern vLLM and FlashAttention on SM70 practical. v100-skinny adds the QPN2/QPN8 execution architecture, the native mixed-checkpoint loader/dispatch path and the SM70 serving fixes described in the repo.

submitted by /u/Simple_Library_2700
[link] [comments]