vLLM vs NInfer — Qwen3.8-27B benchmark (RTX 5090): throughput sweeps (cold/warm) + 69-scenario tool-call suite, tool-eval-bench

README.md

Qwen3.8-27B on RTX 5090 — vLLM vs NInfer

Client-side benchmark of two inference engines serving the same model (Qwen3.8-27B, 262 K context) on a single NVIDIA GeForce RTX 5090 32 GB, measured with tool-eval-bench (v2.5.1.dev30+gded5b8f04, Python 3.14.7): a llama-bench-style pp/tg throughput sweep (depths 0–32 K, concurrency 1/2/4) and a 69-scenario tool-call quality suite.

Setup

Both benchmarks ran at temperature 0.0, seed 42 (deterministic re-runs). Throughput numbers are client-observed effective rates (tokens ÷ wall time, including prefill); see Caveats before reading the tables.

1. Throughput — effective generation rate (client tg t/s)

Identical config: PP=2048 / TG=128, depths 0/4K/8K/16K/32K. At concurrency N the tool fires N parallel requests of 128 tokens each, so c2 = 2×128 and c4 = 4×128 aggregate tokens. vLLM warm = server restarted with --enable-prefix-caching.

† NInfer was capped at --max-concurrency 2: its c4 points measure admission queueing, and the 32 K×c4 point lost 1 of 4 requests (degraded — treat as invalid). vLLM (--max-num-seqs 16) batched all c4 points cleanly.

Update — NInfer NVFP4 (c1). The tables below benchmark the original groupwise-int qwen3_8_27b.ninfer artifact. After installing the upstream NVFP4 artifact (qwen3_8_27b_nvfp4.ninfer, same model + MTP@3 + int8 KV) the c1 single-stream numbers jump and the vs-vLLM verdict flips at short context: NInfer leads @0 (127.1 vs 117.3) and is within noise at 4 K (87.1 vs 87.5), then vLLM pulls ahead with depth (+5 % @8 K, +26 % @16 K, +35 % @32 K).

Context	NInfer NVFP4 c1	vLLM MTP@3 c1
0	127.1	117.3
4 096	87.1	87.5
8 192	73.1	76.7
16 384	44.3	56.0
32 768	24.2	32.7

One caveat: with NVFP4 + MTP@3 the per-request context ceiling drops to 131 072 (128 K) (upstream registers 262 144 for MTP0, 131 072 for MTP3), so a full-262 K NInfer comparison requires MTP off or the groupwise-int artifact. Scope: these NVFP4 numbers are single-stream c1 only (0–32 K), run against a prefix-reuse-enabled server (d0 fully cold; deeper c1 points may get a small shared-base-prefix hit). NVFP4's c2/c4, real prefill, tool-call quality and >32 K context were not re-measured — those all still come from the groupwise-int run below.

NInfer warm run used a different config (PP=4096/TG=256, c1/c2 only, depths 0/4K/16K) — not 1:1 comparable:

What prefix caching changes on vLLM (c1 → c2/c4 reuse of the same prompt; c1 points barely move because each depth prompt is used for the first time):

Server-side reference (per-request timings, single stream):

vLLM + MTP — effective rates and acceptance

--speculative-config '{"method": "mtp", "num_speculative_tokens": N}' (+ APC + froggeric; --max-model-len -1 → 227 200). The MTP head is a single cycled layer (mtp.layers.0.*), so the chain length is a vLLM implementation knob.

Acceptance (spec-bench MTP@3): per-position p0/p1/p2 = 78/60/45 % (~1.83 tok/round; NInfer+MTP 2.0–2.8). Content-bound, not context-bound (the spec-bench acceptance re-measurement reports higher values — α 92.2 % code / 93.1 % structured / ~67–70 % filler, ~2.8 tok/round):

num_speculative_tokens: 4 crashes vLLM 0.27.1 (CUDA illegal memory access in FlashInfer during spec-decode scheduling at 4 streams — not a KV OOM, not a weight limit). 3 is the maximum usable window; the only case where MTP loses is 32 K×c4 (verification cost > draft gain).

2. Tool-call quality — standard suite

69 deterministic multi-turn scenarios across 15 categories (pass=2 / partial=1 / fail=0, 138 max points). Same 52-tool overhead (~4,742 tokens) for the full suite.

Scores

‡ Adjusted for a documented grader false negative on TC-35 (see below): raw scored output is 96/100 (132/138), the adjusted reading is 97/100 (134/138).

Category-level delta (full suite, NInfer vs vLLM + froggeric)

Scenario-level delta (full suite)

Difficulty tiers:

TC-14: the stock-template miss the froggeric template fixes

Short suite, error-recovery scenario: get_stock_price returns a rate-limit error. Stock template: the model explains the failure and suggests retrying / checking manually, never tries an alternative tool (partial). Froggeric-v22: retries get_stock_price, falls back to web_search, surfaces the price — pass. The template teaches the multi-tool error-recovery turn pattern.

TC-35: the grader false negative (‡)

"Convert 500 degrees Kelvin to Kelvin" — a tautology. The vLLM answer is substantively correct (states "500 K = 500 K, no conversion needed"), but adds "for reference" °C/°F lines. The scorer's pass condition is identity-phrase AND "fahrenheit" not in answer — the substring guard meant to catch models that answer 440.33 °F also fires on a mere mention, so a correct-but-thorough answer scores 0/2. Reproduced deterministically (temp 0.0, seed 42). This gist counts TC-35 as a pass (‡); the raw scored run stays in the local artifacts.

Key takeaways

1. Prefill is vLLM's advantage: ~6–18 K t/s vs ~3.0–3.3 K t/s cold NInfer → vLLM leads from 8 K context on up (c1: +57 % @8K, +91 % @16K, +116 % @32 K effective tg).
2. vLLM+MTP@3 is at/under NInfer at short context, ahead with depth: 117.3 vs NInfer NVFP4's 127.1 t/s @ d0 c1 (−8 %), within noise at 4 K (87.5 vs 87.1), then vLLM pulls ahead from 8 K (76.7 vs 73.1 @8K, 56.0 vs 44.3 @16K, 32.7 vs 24.2 @32 K). ~153 t/s pure decode on code content (α 92.2 % via spec-bench) vs NInfer+MTP's 105–150 t/s mixed. On raw tok/round the two are level (NInfer 2.0–2.8 vs vLLM's measured ~2.8).
3. Prefix caching multiplies vLLM at depth: at 32 K the effective tg rises 4.4× (c2) / 7.1× (c4) with --enable-prefix-caching — the realistic number for multi-turn agentic workloads that share context; the bottleneck moves from prefill to decode.
4. Tool-call quality: vLLM + froggeric-v22 wins 97 vs 89 (‡) on the full suite; the froggeric template fixes NInfer's structured-output weakness (12/12 vs 5/12) and the TC-14 error recovery. The one new regression (TC-35) is a grader false negative, not a model error.
5. Deployment: vLLM+MTP@3+APC is the best overall (long-context prefill + 94–154 t/s decode); the one case where MTP loses is 32 K×c4-class batching, where plain warm vLLM (223.2) beats MTP@3 (169.2). vLLM needs explicit flags to match NInfer's defaults (prefix caching is off by default for hybrid models).
6. MTP is content-bound, capped at 3 drafts: α 92.2 % (code) / 93.1 % (structured) / ~67–70 % (filler) via spec-bench; num_speculative_tokens: 4 crashes vLLM 0.27.1 (FlashInfer illegal memory access) — 3 is the max.

Caveats

• Client pp t/s and TTFT are first-SSE-event artifacts in both engines (both emit an initial SSE event on request acceptance, so TTFT is a few ms and pp t/s is hundreds of K). Use the tg t/s columns for engine comparison; real prefill/decode rates are the server-side reference table above.
• Client tg t/s at concurrency 1 includes the prefill wall time of the request (effective end-to-end rate), which is why it falls from ~73 t/s (0 ctx) to ~24 t/s (32 K) even though pure decode is much faster.
• NInfer c4 points include admission queueing (server cap 2); the 32 K×c4 point lost a request and is invalid (†).
• The vLLM warm sweep is mixed by design: c1 points are effectively cold (first use of each prompt), c2/c4 points benefit from full prefix hits of the same prompt.

Files

Reproducing

# throughput — cold: server WITHOUT --enable-prefix-caching
./bench_throughput.sh
# throughput — warm: restart the server WITH --enable-prefix-caching, then
./bench_throughput.sh

# tool-call quality (restart the server with the froggeric chat template
# for the 100/100 short / 97 ‡ full numbers)
./bench_standard.sh --short
./bench_standard.sh

Server (vLLM, final config):

vllm serve gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \
  --quantization modelopt \
  --chat-template /qwen38-froggeric-v22.jinja \
  --kv-cache-dtype fp8 --trust-remote-code \
  --max-model-len -1 --max-num-seqs 16 --gpu-memory-utilization 0.97 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --enable-prefix-caching
# drop --speculative-config for the no-MTP reference numbers;
# drop --enable-prefix-caching for the "cold" numbers

bench-scripts-README.md

vLLM benchmark scripts

Two wrappers around tool-eval-bench targeting the vLLM serve on http://127.0.0.1:8000 (vllm serve gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090, NVFP4 weights, fp8 KV cache, 262 144 max-model-len, --max-num-seqs 16, --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3).

Same design as ../bench (NInfer) so the two reports are directly comparable.

Both scripts:

• benchmark against an already-running vLLM server on $BASE_URL (health-checked via /health); the server is externally managed and is never started or stopped by the scripts (unlike ../bench, which auto-launched NInfer) — START_SERVER=1 + VLLM_LAUNCHER= is supported but off by default
• auto-detect the model id from /v1/models if MODEL is not set
• save tool-eval-bench reports to runs/ (SQLite history in the tool's own data/benchmarks.sqlite)
• pass extra args through to the CLI, e.g. ./bench_standard.sh --short

Usage

# throughput sweep (pp2048/tg128, depths 0–32K, concurrency 1/2/4)
./bench_throughput.sh

# lighter sweep
DEPTH="0,4096,16384" CONCURRENCY="1,2" PP=4096 TG=256 ./bench_throughput.sh

# standard 69-scenario tool-call suite
./bench_standard.sh

# quick 15-scenario check
./bench_standard.sh --short

# hard mode (84 scenarios)
./bench_standard.sh --hardmode

Useful env vars (all optional): BASE_URL, MODEL, OUTPUT_DIR, LABEL, START_SERVER=1+VLLM_LAUNCHER, SERVER_WAIT; throughput: PP, TG, DEPTH, CONCURRENCY, TIMEOUT; standard: SEED, TRIALS, PARALLEL, TIMEOUT.

Notes

• Throughput uses the built-in engine (--perf-legacy-only) to match the NInfer run's methodology (same client-side pp/tg/TTFT). vLLM ships a HF tokenizer, so ./bench_throughput.sh --perf (llama-benchy) is also available if you want the external-benchmark flavour.
• Prefix caching: vLLM (V1) has automatic prefix caching on by default and the sweep reuses the same prompt across points, so pp t/s benefits from KV cache hits after the first point (the script prints a note). For cold-cache numbers restart the server with --no-enable-prefix-caching and re-run; check live hits with curl -s http://127.0.0.1:8000/metrics | grep vllm:prefix_cache_hits_total. Observed on 2026-08-17: with the default flags (no explicit --enable-prefix-caching) the prefix_cache_queries/hits counters stayed at 0 through the whole sweep — vLLM 0.27.1 keeps APC opt-in for hybrid (linear-attention) models (this is a Qwen3_5 VLM, hybrid linear + full attention), so the default is off — those numbers were already effectively cold. After adding --enable-prefix-caching the counters work (72.7 % hit rate on the sweep) and c2/c4 points, which reuse the same prompt as the c1 point of each depth, jump ~4–7× in effective tg t/s at deep context (32768: c2 28→124, c4 31→223 t/s).
• Client pp t/s / TTFT are artifacts (same as the NInfer report): vLLM emits its first SSE event on request acceptance, so the client's TTFT (a few ms) and pp t/s (hundreds of K) do not reflect the real prefill time (measured ~260 ms for a ~4 K-token prompt on this server). Use them for the tg sweep shape, and cross-check real prefill/decode against the server logs or a raw HTTP timing.
• The server is capped at --max-num-seqs 16, so all sweep concurrency points (1/2/4) are batched without admission queueing (the NInfer run was capped at 2, its c4 points measured queueing).
• The scenario path uses the native --backend vllm label (no hacking, unlike the NInfer run which had to use llamacpp).
• Tool calls are parsed natively (--enable-auto-tool-choice --tool-call-parser qwen3_xml); thinking content goes into the separate reasoning field and stays out of the parser.

Reproducing the vLLM server

cd ~/src/vllm
.venv/bin/vllm serve gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \
  --quantization modelopt --kv-cache-dtype fp8 --trust-remote-code \
  --max-model-len 262144 --max-num-seqs 16 --gpu-memory-utilization 0.97 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml

bench_common.sh

bench_standard.sh

bench_throughput.sh

ninfer-benchmark-report.md

NInfer — qwen3.8-27b Benchmark Report

Date: 2026-08-17 Host: HoloDECK — Linux 7.1.8-1-cachyos-x86_64 (CachyOS), NVIDIA GeForce RTX 5090 32 GB (driver 610.57.04) Engine: NInfer serve (ninfer-serve), OpenAI-compatible API on http://0.0.0.0:8080 Harness: tool-eval-bench v2.5.1.dev30+gded5b8f04 (Python 3.14.7)

Setup

Server launched from ~/src/ninfer with the NInfer launcher:

cd ~/src/ninfer
./start.sh models/qwen3_8_27b.ninfer

which resolves to:

./build/apps/ninfer-serve models/qwen3_8_27b.ninfer \
  --host 0.0.0.0 --port 8080 \
  --max-context 262144 --kv-capacity auto --max-request-mib 2048 \
  --kv-dtype int8 --max-concurrency 2 \
  --spec mtp --draft-tokens 3 --lm-head-draft \
  --preserve-thinking \
  --temperature 1.0 --top-k 20 --top-p 0.95 --min-p 0.0 --presence-penalty 0.0

Highlights: 256 K context, int8 KV cache (auto-sized, 366 K tokens free after weights), MTP speculative decoding (3 draft tokens + LM head), thinking preserved, sampler at llama.cpp-parity values.

Weights profile: the original qwen3_8_27b.ninfer artifact (18.2 GB) is the groupwise-int profile (per the file's weights_id). The upstream NVFP4 artifact (qwen3_8_27b_nvfp4.ninfer, 21.5 GB, weights_id: nvfp4) is now installed and benchmarked: c1 single-stream is 127.1/87.1/73.1/44.3/24.2 (0/4/8/16/32 K) vs groupwise-int's 85.2/52.4/34.3/21.1/11.3 — a large jump, and enough to flip the short-context verdict against vLLM. All tables below are the original groupwise-int run.

The throughput benchmark was run against a cold-cache server started by the bench script itself with a launcher variant that adds --no-prefix-reuse (prefix caching off, so every request is a real prefill — confirmed per-request in the server log: reuse=full_reset on every line). The standard tool-call runs reused the plain start.sh server; tool-call scoring is unaffected by prefix reuse.

Benchmark commands, run from ~/src/bench:

cd ~/src/bench
./bench_throughput.sh
DEPTH="0,4096,16384" CONCURRENCY="1,2" PP=4096 TG=256 ./bench_throughput.sh
./bench_standard.sh --short
./bench_standard.sh

1. Throughput

LLaMA-bench-style streaming sweep (TTFT, prompt/generation token rates), measured client-side over SSE and cross-checked against the server's own per-request timings.

How to read the table. At concurrency N the tool fires N parallel requests, each generating TG=128 tokens, and reports the aggregate volume (N×128) — so tg256 @ c2 = two parallel 128-token streams, tg512 @ c4 = four. The tg t/s column is client-observed end-to-end throughput (tokens ÷ wall time), so at long contexts it includes the deep prefill latency (see Caveats). The pp t/s column is a client-side SSE artifact and is not real prefill throughput (see Caveats).

Primary run — PP=2048 TG=128, depths 0/4096/8192/16384/32768, concurrency 1/2/4, cold cache

(run ID 2026-08-17T06-59-08Z_9ee5d4a9)

¹ Client-side pp t/s is an artifact: the client clocks TTFT on the first SSE chunk that arrives immediately (2–37 ms), not on completion of the real prefill (hundreds of ms to seconds). Real prefill is ~3.0K–3.3K t/s (next table). ² Client-observed end-to-end tokens/s including the prefill phase of the request — it is not pure decode speed; pure decode is ~105–150 t/s (next table). ³ Client-side TTFT artifact, same cause as ¹. † At depth 32768 × c4 one of the four concurrent requests was rejected server-side (error inference request expired while waiting for admission — the server caps concurrency at 2 and the queued request outlived the admission budget). Only 3 of 4 streams contributed, so this point is degraded (tokens 23127+384) — treat it as invalid.

Server-side reference metrics (per-request, from the server log; cold cache, reuse=full_reset)

Real cold prefill: ≈ 3.0–3.3 K token/s (drops slightly at 32 K). Pure decode: ≈ 105–150 t/s single stream, essentially flat across context length; MTP acceptance 43–61 % with ~2.0–2.8 tokens/round.

Secondary sweep — PP=4096 TG=256, depths 0/4096/16384, concurrency 1/2

(run ID 2026-08-17T06-40-16Z_2c7dda88; client numbers on the earlier prefix-reuse server — tg t/s valid, pp client metrics inflated; run for reference of the different config)

Caveats

• Client pp t/s is not a prefill measurement. The client's TTFT is the time to the first SSE chunk, which NInfer emits immediately; the real prefill takes 0.6–10 s for 2K–31K tokens on this GPU (server TTFT, table above). Client pp t/s figures (450K–910K) should be ignored; use the server-side prefill column instead.
• Client tg t/s at c1 includes the deep prefill wall time, which is why it falls from 85 t/s (0 ctx) to 11 t/s (32 K) even though pure decode is flat at ~105–150 t/s. As a metric it is "effective end-to-end generation throughput", not decode speed.
• The server is capped at --max-concurrency 2; c4 points measure admission-queueing as well as batching, and the deepest c4 point lost one request to the admission budget (†).
• Token counts match the server exactly (e.g., 1879, 5491, 9111, 16355, 30836), so the client calibration is accurate.

2. Tool-Call Quality — Standard Suite (tool-eval-bench)

69 deterministic multi-turn scenarios across 15 categories (12 universal tools, 52 tools for Category L), scored pass=2 / partial=1 / fail=0. Requests ran at temperature 0.0 (per-request override), thinking mode enabled, seed 42.

Quick run — core 15 scenarios (--short)

(run ID 2026-08-17T06-40-38Z_b3c5f815)

Score: 100 / 100 — ★★★★★ Excellent (30/30 points, 15/15 passed, deployability 96, median turn 1.0 s)

Full run — all 69 scenarios

(run ID 2026-08-17T06-41-25Z_475e20c2)

Score: 89 / 100 — ★★★★ Good (123/138 points · deployability 87, α=0.7 · quality 89 · responsiveness 83, median turn 1.1 s · tool-definition overhead ≈ 4,742 tokens for the 52-tool set)

Failing / partial scenarios

Key takeaways

1. Tool-calling quality is strong: 100% on selection, parameters, chains, restraint, recovery, localization, code patterns; 96% on safety; full quick suite 100/100.
2. Structured output is the weak spot (Category O: 42%): in most cases the model performs the correct tool calls but replies with prose instead of schema-valid JSON — a chat-template/instruction-tuning issue, not an orchestration failure.
3. Long-horizon multi-turn degrades (Hard tier: 65%): deep research chains and goal-level planning lose state or overrun the turn budget; safety behaviour (injection, refusal, authority escalation) holds at 96%.
4. Prefill is ~3.0–3.3 K t/s cold (RTX 5090), dropping to ~2.95 K t/s at 32 K context; TTFT 0.6 s (0 ctx) → 10.5 s (32 K).
5. Decode is ~105–150 t/s single stream with MTP (2.0–2.8 tok/round, 43–61% acceptance), flat across context; end-to-end effective generation drops with depth because it includes the deep prefill.
6. Concurrency is maxed at 2 by the launcher; 4-way points queue on admission and the 32K×c4 point lost a request to the admission budget.

Reproducing

# standard runs (server: prefix reuse on)
cd ~/src/ninfer && ./start.sh models/qwen3_8_27b.ninfer

# benchmarks (throughput auto-starts a cold-cache server variant and stops it)
cd ~/src/bench
./bench_throughput.sh
DEPTH="0,4096,16384" CONCURRENCY="1,2" PP=4096 TG=256 ./bench_throughput.sh
./bench_standard.sh --short
./bench_standard.sh

Artifacts (per-scenario traces included): ~/src/bench/runs/2026/08/*--ninfer-throughput.md, ~/src/bench/runs/2026/08/*--ninfer-standard.md; server per-request logs in ~/src/bench/logs/; SQLite history in ~/src/bench/data/benchmarks.sqlite.

vllm-benchmark-report.md

vLLM — Qwen3.8-27B-NVFP4-RTX5090 Benchmark Report

Date: 2026-08-17 Host: HoloDECK — Linux 7.1.8-1-cachyos-x86_64 (CachyOS), NVIDIA GeForce RTX 5090 32 GB Engine: vLLM 0.27.1 (/home/pier/src/vllm), OpenAI-compatible API on http://127.0.0.1:8000 Model: gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 (NVFP4 weights, fp8 KV, 262 144 max-model-len, --max-num-seqs 16) Harness: tool-eval-bench v2.5.1.dev30+gded5b8f04 (Python 3.14.7)

Setup

Two server variants (externally managed, restarted in between):

# cold — no --enable-prefix-caching (vLLM 0.27.1 keeps APC opt-in for hybrid models;
# counters stayed at 0)
vllm serve gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \
  --quantization modelopt --kv-cache-dtype fp8 --trust-remote-code \
  --max-model-len 262144 --max-num-seqs 16 --gpu-memory-utilization 0.97 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml

# warm — same + --enable-prefix-caching

Runs (client-side tg t/s, --perf-legacy-only, same methodology as ../bench):

Summary — effective generation throughput (client tg t/s)

Identical config PP2048/TG128 (c2 = 2×128, c4 = 4×128 aggregate tokens):

† NInfer c4 points measured admission queueing (server cap --max-concurrency 2); the 32 K×c4 point lost 1 of 4 requests — invalid. vLLM (--max-num-seqs 16) batched all c4 points cleanly.

NInfer warm run was a different config (PP4096/TG256, c1/2 only, depths 0/4K/16K) — not 1:1 comparable with the vLLM sweeps:

What the cache changes (vLLM, c1→c2/c4 reuse of the same prompt)

c1 points barely move (each depth prompt is used for the first time), so the c1 column stays the valid "cold" comparison; c2/c4 warm points measure "cached prefill + decode" — the realistic number for multi-turn agentic workloads that share system prompt / tools / history.

vLLM + MTP (speculative decoding)

Server started with --speculative-config '{"method": "mtp", "num_speculative_tokens": N}' (froggeric chat template + --enable-prefix-caching kept; --max-model-len -1 resolves to 227 200 with MTP@3). The model's MTP head is a single cycled layer (mtp.layers.0.* in the weights; spec_step_idx % num_mtp_layers in qwen3_5_mtp.py), so the draft chain length is an implementation knob, not a weight limit.

Effective generation rate (client tg t/s, PP2048/TG128)

MTP@3 ≥ MTP@1 on every point; c1 gains +3.5 % @d0 to +12 % @4–16K over the warm baseline, and at d0 it's within ~8 % of NInfer NVFP4 (117.3 vs 127.1; the groupwise-int artifact was 85.2). The two exceptions are 32 K×c2/c4, where the batched verification cost still loses to plain decode (c4: 169.2 vs 223.2) — noise range for a 1-sample-per-point sweep is roughly ±5–10 %, so the small deltas (e.g. MTP@1 c2 @d0 118.4 vs warm 133.6) are not significant.

Acceptance (spec-bench MTP@3, run 2026-08-17T09-48-18Z_2e9d6016)

Per-position acceptance from server metrics (sweep aggregate): p0 78.1 %, p1 60.0 %, p2 44.6 % → ~1.83 tok/round (MTP@1: 1.81). NInfer+MTP: 2.0–2.8 tok/round (43–61 % acceptance, 3 drafts). (A dedicated spec-bench re-measurement reports higher acceptance — α 92.2 % code / 93.1 % structured / ~67–70 % filler, ~2.8 tok/round — than the sweep aggregate below.)

The acceptance rate is content-bound, not context-bound: code and structured keep identical α across depths 0–8 K, and code's pure decode is ~153 t/s flat — the highest single-stream decode of any configuration (NInfer+MTP mixed: 105–150 t/s). The throughput sweep's prompts are filler-like, which is why its c1 numbers track the filler row (117.3 vs 95.1 eff t/s @d0). Draft window utilisation: 2.2/3 (75 %); the third position (α 44.6 %) is the wasteful one.

num_speculative_tokens: 4 crashes vLLM 0.27.1

Attempted @4 (DEPTH=0,4096,32768 CONCURRENCY=1,2,4): the d0 c1 point measured 109.1 t/s (+15 % vs @3), then the engine died at d0 c4 with torch.AcceleratorError: CUDA error: an illegal memory access in the FlashInfer attention-metadata build during spec-decode scheduling (4 streams × 4 draft positions). Not a KV OOM (kv_cache_usage 45 %) and not an architectural limit (the head cycles its single layer) — an upstream vLLM 0.27.1 bug in the MTP/eagle spec-decode path. 3 is the maximum usable window in vLLM 0.27.1 (proven stable through two full sweeps + spec-bench). Side effect seen at startup with @4: max_seq_len drops to 213 312 (from 227 200) because the wider draft window reserves more KV for the head.

Config recommendation: num_speculative_tokens: 3 + --enable-prefix-caching

• froggeric template is the best overall; disable MTP only for 32 K×c4-class workloads, where the verification cost still loses to plain decode.

Key takeaways

1. Prefill is vLLM's advantage: ~6–18 K t/s (4.7 K prompt in ~260 ms) vs NInfer cold ~3.0–3.3 K t/s → vLLM wins from 8 K context on up (c1: +57 % @8K, +91 % @16K, +116 % @32K).
2. vLLM+MTP@3 is at/under NInfer at short context, ahead with depth: with --speculative-config mtp/3 the d0 c1 rate goes 72.0 → 117.3 t/s (−8 % vs NInfer NVFP4's 127.1; the older groupwise-int artifact was 85.2); on code-like content pure decode reaches ~153 t/s (α 92.2 % via spec-bench) vs NInfer+MTP's 105–150 t/s mixed. Raw tok/round is level (NInfer 2.0–2.8 vs vLLM's measured ~2.8).
3. Prefix caching multiplies vLLM at depth: at 32 K the effective tg rises 4.4× (c2) / 7.1× (c4) when the prompt is cached — the bottleneck moves from prefill to decode.
4. vLLM 0.27.1 keeps prefix caching opt-in for hybrid (linear-attention) models — this model is a Qwen3_5 VLM (Qwen3_5ForConditionalGeneration, hybrid linear + full attention), so the default is off; --enable-prefix-caching must be passed explicitly (72.7 % block hit rate on the warm sweep).
5. Client pp t/s / TTFT are first-SSE-event artifacts in both engines — ignore them; the tg t/s columns above are the comparable metric.
6. Tool-call quality: vLLM + froggeric-v22 wins 97 vs 89 ‡ on the full suite; the froggeric template fixes NInfer's structured-output weakness (12/12 vs 5/12). TC-35 is scored as a pass after a documented grader false-negative correction (raw: 96).
7. MTP is content-bound and capped at 3 drafts in vLLM 0.27.1: α 92.2 % on code / 93.1 % structured / ~67–70 % on filler (~2.8 tok/round via spec-bench; sweep-aggregate p0/p1/p2 = 78/60/45 %, 1.83 tok/round); num_speculative_tokens: 4 crashes the engine (FlashInfer illegal memory access) — 3 is the max, and 32 K×c4-class workloads still run faster without MTP.

Tool-call quality — standard suite

vLLM short runs (tool-definition overhead ~1,158 tokens, 12 tools):

• stock template, run 2026-08-17T08-06-38Z_1917ad7c: 97/100, deployability 90, median turn 1.6 s. Single partial: TC-14 Malformed Response — after the get_stock_price rate-limit error the model explained the failure and suggested retrying / checking manually, but never tried an alternative tool. NInfer passed this scenario on its short run.
• qwen38-froggeric-v22.jinja, short run 2026-08-17T08-13-13Z_484c3fa4: 100/100 (30/30), deployability 93, median turn 1.4 s. TC-14 now passes: the model retries get_stock_price, then falls back to web_search and surfaces the price ($187.42) — the exact recovery behaviour the stock template missed. Parity with NInfer on the short suite.

Full suite — vLLM + froggeric (97/100 ‡) vs NInfer (89/100)

vLLM full run 2026-08-17T08-15-38Z_0f81cdfe: deployability 88 (α=0.7), quality 96 ‡, responsiveness 68 (median turn 1.8 s; NInfer 1.1 s → 83), tool-definition overhead 4,742 tokens (52 tools).

‡ Adjusted for the TC-35 grader false negative documented below (23/26 → 25/26; 132/138 → 134/138; 96 → 97). Raw tool output: 0f81cdfe scored 96/100 (132/138).

NInfer's signature weakness (structured output, 42 %) is fully fixed by the froggeric template: TC-64/65/66/67/68/69 all pass with valid JSON. Scenario-level delta: TC-45 and TC-64 fail→pass, TC-51 fail→partial, TC-49/65–69 partial→pass, offset by TC-35 (Contradictory Parameters) now failing — flagged safety-critical: the model answered in the wrong unit instead of recognizing the Kelvin identity (NInfer passed it). Remaining partials (TC-46/57/62 — long-horizon chains) are the same set NInfer had, plus TC-51 as partial instead of fail.

Difficulty tiers: vLLM 4/4 · 17/17 · 30/31 ‡ · 13/17 (hard 76 %) vs NInfer 4/4 · 15/17 · 27/31 · 11/17 (hard 65 %) — ‡ moderate becomes 31/31 with the TC-35 adjustment.

TC-35 investigation (the only new failure)

Reproduced deterministically (temp 0.0, seed 42, single-scenario re-run — identical answer). The model's answer is actually correct:

**500 K = 500 K** — No conversion is needed! "Degrees Kelvin" and
"Kelvin" refer to the same unit ... For reference, 500 K is equivalent
to: 226.85 °C / 440.33 °F

Root cause is the grader's crude guard in scenarios_agentic.py (_tc35_eval): the pass condition requires identity-phrase AND "fahrenheit" not in answer. The "fahrenheit" substring guard (meant to catch models that answer 440.33 °F) also fires on answers that merely mention °F as a bonus reference line, so the correct-but-thorough answer is graded 0/2 while a terse "no conversion needed" answer (what NInfer produced) passes. This is a grader false negative, not a model error: vLLM's effective quality on TC-35 is pass-level. Per that finding this report scores TC-35 as a pass (‡, 134/138 → 97/100); the raw tool output (96/100, 132/138) stays in the run artifacts (runs/, data/benchmarks.sqlite) for traceability. Candidate upstream fix: only fail when the °F value is the primary answer (e.g. check the first line / the stated result), not on any "fahrenheit" mention.

Reproducing

# throughput (cold: server without --enable-prefix-caching)
cd ~/src/vllm_bench
./bench_throughput.sh          # -> fab45c6e-style numbers

# throughput (warm: restart server with --enable-prefix-caching)
./bench_throughput.sh         # -> 11c87a8e-style numbers

# throughput (best: + --speculative-config '{"method":"mtp","num_speculative_tokens":3}')
./bench_throughput.sh         # -> 6b90eb98-style numbers

# acceptance detail per prompt type (run against the MTP@3 server)
tool-eval-bench bench --spec-bench --spec-method mtp --skip-tool-eval \
  --backend vllm --format openai --no-probe-engine \
  --base-url http://127.0.0.1:8000 --model "" \
  --label vllm-spec-mtp3 --output-dir runs

# tool-call quality (restart the vLLM server with the froggeric chat
# template for the 100/100 short / 97 ‡ full numbers)
./bench_standard.sh --short   # stock template -> 1917ad7c (97); froggeric-v22 -> 484c3fa4 (100)
./bench_standard.sh           # full 69-scenario suite -> 0f81cdfe (96/100 as scored; 97 ‡ adjusted, froggeric template)

Artifacts: ~/src/vllm_bench/runs/2026/08/*--vllm-throughput.md, SQLite history in ~/src/vllm_bench/data/benchmarks.sqlite.