I pushed Qwen3.8-27B to 381 tps for a single request on a RTX 3090

Four days ago I released a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090 (82 tps single request, 672 peak). Since then it went to ~114, then ~138 tps single-user with DFlash2 drafting and lookup-augmented drafting. Today it's ~133 tps on real chat prompts, 382 tps when the model reproduces its own context, and the number I care about most this time: a document-quoting workload now runs at 15 of 16 tokens accepted per verify step. What we had: - fp8 KV cache, lm_head + embed_tokens int8, fp16 recurrent state, int8 activations, DFlash2 block drafting (W4A16-requantized), lookup-augmented drafting, prefix caching for the hybrid, split-KV verify attention, sampler patch, KVarN for 262k context Now added: - Longer verify blocks. DFlash2 only drafts 7 tokens and I'd assumed the verify block had to match. It doesnt. The lookup drafter fills the extra slots from the request's own context for free. DFLASH_TOKENS=15 verifies 16 tokens per step: 260 to 382 tps reproducing a 25k-token document, +9% on ordinary chat. Costs half the request slots and 8k of context, so it's opt-in. - DFlash2 past 64k. bf16 KV is 64 KB/token and teh pinned pool is 5.2 GiB, which is what capped it. An int8 cache roughly doubles it, 138,696 tokens instead of 69,758. Two fixes were needed: vLLM equalises KV page sizes by an integer block ratio, and the drafter's 5 sliding-window layers were wasting 5.2 GiB at 1.88% utilisation until I rounded their block size from 16 to 864. - Honest result: +53% where the model reproduces its context, about 2:1 behind plain MTP everywhere else, with twice the TTFT. So it's a mode for RAG front-ends and coding assistants applying edits, not a default. - A correction. Someone opened an issue saying our long-context numbers didn't reproduce. They were right. The tables were measured in the batch config, which runs no speculative decoding, and nothing said so. The KVarN decode tax we documented as ~20% is 2.13x single-user at 112k. Most of that is step time, the rest is MTP acceptance falling from 2.56 to 2.38 tokens per step, becuase the quantised cache moves the target's logits enough that the draft head agrees less often. Quality-neutral doesn't mean speed-neutral once you're speculating. Quality unchanged throughout, GSM8K 96.5%. int8 KV costs 2x the prefill, so it's for loading a document once and asking many quesitons about it. All of it is one env var apart. Quality unchanged throughout (GSM8K 96.5%), and speculative decoding is exact by construction. Caveats worth stating: the long verify block is worth it when your output quotes the prompt and roughly neutral when it doesn't, and int8 KV costs 2× the prefill of bf16, so it's for workloads that load a document once and then ask many questions about it. All of it is one env var apart. Repo: github.com/syv-ai/qwen38-27b-rtx3090 I've said "probably the last update" twice now. Since then two people sent PRs, one of them answered a question I'd assumed was a dead end, and a third found a number I'd got wrong. Turns out publishing what didn't work gets you better bug reports than publishing what did. Also, I have been considering doing this for 4090's and 5090's too, since they have a few options that 3090's dont. So if anyone has a spare 4090 or 5090 or wants to rent me one on runpod/vast, let me know.