Qwen3.8-Flash-Next NVFP4 2xDGX Spark config: 50t/s decode, 2,900t/s prefill

After a couple of days of faffing around, here's my current config in case it helps anyone out. Some of this is also valuable for a single Spark. Benchmarks: Dual-Spark TP2, eager + MTP k=3, 262k context (warmed medians, exact tokenizer counts): ┌─────────┬──────────────────┬───────────────────┬─────────────────────────────────┐ │ Streams │ Decode aggregate │ Decode per-stream │ Prefill aggregate (10k prompts) │ ├─────────┼──────────────────┼───────────────────┼─────────────────────────────────┤ │ 1 │ 45.9 t/s │ 45.9 t/s │ 2,940 t/s │ ├─────────┼──────────────────┼───────────────────┼─────────────────────────────────┤ │ 4 │ 120.2 t/s │ 30.0 t/s │ 2,524 t/s │ ├─────────┼──────────────────┼───────────────────┼─────────────────────────────────┤ │ 8 │ 222.2 t/s │ 27.8 t/s │ 3,098 t/s │ ├─────────┼──────────────────┼───────────────────┼─────────────────────────────────┤ │ 13 │ 265.9 t/s │ 20.5 t/s │ 2,960 t/s │ └─────────┴──────────────────┴───────────────────┴─────────────────────────────────┘ Prefill vs prompt depth (single stream, exact tokens): ┌─────────────┬───────────┐ │ Prompt size │ Prefill │ ├─────────────┼───────────┤ │ 11k │ 2,875 t/s │ ├─────────────┼───────────┤ │ 100k │ 2,655 t/s │ ├─────────────┼───────────┤ │ 200k │ 2,463 t/s │ └─────────────┴───────────┘ AI slop below for you to read and paste into your own agent :) Stack vLLM PR #53896 ( release/qwen38next branch) — NOT main; main doesn't have the model, and the recipe image's vLLM commit isn't in the public repo sm_121 support is a 2-file patch (details below) — GB10 is arch 12.1 and the NVFP4 E2M1 conversion needs a software fallback PLE n-gram table served from internal NVMe via mmap:(48 GB, MADV_RANDOM is essential — it's a 30× read-amplification difference on hash-scattered row lookups), with a node-local CPU-worker process doing gathers over ZMQ + pinned buffers + CUDA-IPC outputs so the gather is graph-safe and TP2-safe TP2 across both Sparks: native venv (NOT docker — see dead ends), eager + MTP k=3 Numbers (TP2, warmed medians, count-20 structured / 3-paragraph prose) config structured prose prefill (11k tokens) eager + MTP k=3 49.7 34.8 ~2,875 Dead ends PIECEWISE cudagraphs cost ~28% decode under MTP+TP2. Same tree, same everything, graphs on vs off: 36 vs 50 tok/s. Graphs help single-node no-MTP decode (+80%, 9→16.5), but combined with MTP at TP2 they're a straight loss — MTP already amortizes the launch overhead the graphs were eliminating. We chased a "tree-level MTP regression" for a day before realizing the config was the regression. Docker silently degrades NCCL to TCP sockets. Default containers get no IB device → NET/Socket → TP2 prefill at ~40% of native. You need --privileged (or IB device passthrough) AND the right HCA name — see next point. Native venv runs just get RoCE. RoCE device names are not stable across reboots, and can differ between two identical machines (ours: rocep1s0f1 on one, rocep1s0f0 on the other, resolved dynamically). Hardcoded NCCL_IB_HCA will silently fail on one node. Enumerate with ibdev2netdev , match port-ACTIVE to your rail netdev, and verify with NCCL_DEBUG=INFO that you see NET/IB , not NET/Socket . flashinfer must be 0.6.18 on GB10 — 0.6.17 crashes the NVFP4 MoE fallback kernel. vLLM's MoE shuffleInputRowsKernel has an uninitialized-permutation OOB read that shows up as a fake CUTLASS status=7 GEMM failure — a two-line bounds guard fixes it (already in PR #53896). The 2-file sm_121 patch (vs PR #53896 base) CMakeLists.txt : add 12.1 to CUDA_SUPPORTED_ARCHS , SCALED_MM_ARCHS , CUTLASS_MOE_DATA_ARCHS , FP4_SM120_ARCHS csrc/libtorch_stable/quantization/fp4/nvfp4_utils.cuh : ~90-line software E2M1 conversion guarded by __CUDA_ARCH__ == 1210 (GB10 has no cvt.rn.satfinite.e2m1x2.f32 ) That's the whole delta. Build offline with the usual dependency cache; CUDA arch 121. Startup command(s) Head node (rank 0): export VLLM_PLE_MMAP=1 VLLM_PLE_MMAP_WORKERS=64 VLLM_PLE_MMAP_PREWARM=1 export VLLM_PLE_MMAP_DIR=$HOME/ple-table # internal NVMe copy of the table export VLLM_PLE_CPU_OFFLOAD=1 # activates the IPC CPU-worker gather path export NCCL_SOCKET_IFNAME= # bootstrap only; NCCL finds RoCE itself export GLOO_SOCKET_IFNAME= python -m vllm.entrypoints.openai.api_server \ --model \ --host 0.0.0.0 --port 8086 --load-format safetensors \ --tensor-parallel-size 2 --nnodes 2 --node-rank 0 \ --master-addr --master-port 29511 \ --distributed-executor-backend mp \ --max-model-len 262144 --max-num-seqs 13 --gpu-memory-utilization 0.85 \ --no-enable-prefix-caching --enable-chunked-prefill --max-num-batched-tokens 8192 \ --long-prefill-token-threshold 4096 --enforce-eager \ --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' Worker node (rank 1): identical, but --node-rank 1 --headless , launched ~30s after the head. Note --enforce-eager — that's not a compromise, it's the fast path (see dead end #1).