NInfer vs llama.cpp vs vLLM: quality + speed comparison for Qwen3.8-27B NVFP4 on RTX 5090

I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been watching other redditors post their custom configurations, and I wanted to share what I tested to optimize for a single RTX 5090. I was on llama.cpp (Q5_K_M GGUF, q5_1 KV, 262K context, MTP), but it's limited to parallel=1 and I wanted concurrent serving. So I tested vLLM and NInfer as NVFP4 replacements and did a proper quality evaluation instead of just vibes. Hardware - RTX 5090 32GB (eGPU, OCuLink Gen4 x4) (yes, it's in an eGPU dock 😂 but that only affects model loading) - Ryzen 7 7840HS, 32 GB DDR5 - Ubuntu 26.04, nvidia driver 610.43.02 (open) ## Engine configs llama.cpp\ vLLM\ NInfer\ Quant Q5_K_M GGUF NVFP4 NVFP4 KV cache q8_0 FP8 FP8 Context 196K 262K 240K MTP On (gate failed) None MTP3 (76% acceptance) Concurrency parallel=1 Continuous batch x2 lanes VRAM 31.6 GB 29.6 GB 30.5 GB How the eval worked I built a custom harness with 6 tiers, 50 items each, all from my actual production workload (not generic benchmarks): Relevance classification\ - is this industry relevant? (binary, 50 labeled deals) Needle retrieval\ - planted facts in real industry podcast/video transcripts at 64K/128K/192K/240K context Multi-transcript QA\ - questions across 3-4 concatenated diarized transcripts, 120K+ tokens, including unanswerable controls Reasoning with thinking\ - numeric/logic problems, thinking mode on, greedy pass@1 Structured extraction\ - custom extraction prompt, json_mode (skipped on NInfer, it doesn't support json_mode) Tool replay\ - replayed recorded agent episodes against each engine (diagnostic only, all engines fail this one) Everything paired across engines: same prompts, same seeds (42), same gold labels. No cache_prompt. Statistical comparison uses paired cluster-bootstrap CIs with pre-registered non-inferiority margins. And when I do development with Claude Code, I often leverage multi-model consultations for design, planning, and code review. I did a 4-model review panel (Codex/GPT-5, DeepSeek V4 Pro, Grok 4.5, Kimi K3) audit the methodology mid-campaign. They found 10 issues, including 4 mislabeled gold items where the models were actually right and my labels were wrong. Fixed everything and reran. Quality results Tier\ llama.cpp\ vLLM\ NInfer\ Relevance 86.0% 84.0% 86.0% Needle (conditional) 100% (29/29) 100% (41/41) 100% (41/41) Transcript QA 82.0% 78.0% 88.0% Reasoning 100% 100% 98.0% Extraction F1 0.300 F1 0.350 skipped Tool replay 0% all errors 0% Needle counts differ because llama.cpp's 196K context can't fit the 192K items (need room for max_tokens + headroom). NInfer and vLLM both handle 192K fine. All engines score 100% on every needle they can fit. Statistical comparison (NInfer vs llama.cpp, bootstrap): - Needle: delta = -0.29 (NInfer better, p=0.0006) - this is entirely from context capacity, not retrieval quality - Transcript QA: delta = -0.03, p=0.69 - no difference - Reasoning: delta = +0.02, p=0.72 - no difference - Relevance: McNemar p=1.0 - identical - Tool replay: delta = 0.0 - both fail equally Takeaway: quality is statistically indistinguishable across all engines.\ Speed results (perf probe, server-side timings) Metric\ llama.cpp\ NInfer\ Speedup\ Decode 1K\ 114 tok/s 158 tok/s 1.4x Decode 32K\ 109 tok/s 213 tok/s 2.0x Decode 128K\ 72 tok/s 202 tok/s 2.8x\ Prefill 1K 1,545 tok/s 7,265 tok/s 4.7x\ Prefill 32K 2,155 tok/s 6,892 tok/s 3.2x Prefill 128K 1,528 tok/s 3,904 tok/s 2.6x TTFT 1K 670 ms 138 ms 4.9x TTFT 32K 15.2 s 4.8 s 3.2x TTFT 128K 85.9 s 33.6 s 2.6x vLLM speed excluded from the table because the perf probe used wall-clock timing (includes prefill + scheduling + decode) instead of server-side timings, so the numbers aren't comparable. From community reports and my own task-level measurements, vLLM does about 70 tok/s decode at short context, which actually matches NInfer's raw step rate (~66 tok/s). The speed difference is entirely MTP3 speculative decoding. What I learned NInfer's speed advantage is all MTP.\ The raw NVFP4 kernel speed is about the same between NInfer and vLLM (~66-70 tok/s). NInfer's MTP3 speculation with 76% acceptance gets you to 158-213 tok/s. If vLLM or llama.cpp had working MTP on NVFP4, the gap would mostly close. The decode speedup grows with context.\ At 1K context it's 1.4x. At 128K it's 2.8x. MTP acceptance stays high even at long context while llama.cpp's dense decode gets slower as context grows. NInfer's tokenizer endpoint is great.\ It exposes /v1/messages/count_tokens (Anthropic Messages format) which gives exact token counts. No more len(text)//3 heuristics. NInfer does NOT support json_mode (as far as I can tell).\ response_format: json_object returns 400. If you need structured JSON output, you'll need to route those calls elsewhere or use prompt-based enforcement. Don't trust vibes for quality.\ I went in expecting NVFP4 might lose a few points vs Q5_K_M. It didn't. Not on any tier. The biggest delta across 250+ items was 2 percentage points, well within noise at n=50 (SE ~5.6pp). Verdict NInfer NVFP4 replaces llama.cpp as my production engine. Same quality, 1.4-2.8x faster decode, 2.6-4.7x faster prefill, concurrent serving. The only gap is json_mode. I put together a detailed poster with all the charts and methodology details: full results poster (claude.ai/code/artifact/b6041437-2cf6-4198-a722-9b4ce853ccc3) Setup if you want to try it: ` # NInfer (from source) git clone github.com/Neroued/ninfer && cd ninfer mkdir build && cd build cmake .. -DCMAKE_BUILD_TYPE=Release -GNinja && ninja # Model (HuggingFace) # huggingface.co/neroued/Qwen3.8-27B-nvfp4-NInfer (20 GB) # Run ./ninfer-serve /path/to/model.ninfer \ --model-id qwen3.8-27b \ --host 0.0.0.0 --port 8080 \ --max-context 240000 --kv-capacity 240000 \ --max-concurrency 2 --kv-dtype fp8 \ --spec mtp --draft-tokens 3 \ --vision --preserve-thinking `