Flash Next (Q2) on a 24GB M5 MacBook Air: 49 t/s decode with Splash expert streaming

Qwen3.8-Flash-Next (ISTA GSQ-RCO Q2_0) + Flash-Next-DFlash2 draft on a MacBook Air M5 (24 GB), on Splash 1.4.0 with a small fork. Experts that don't fit are streamed from the SSD: 6,800 of 24,576 expert slots (~28%) live in ~8.8 GB, the rest is read on demand, 256K context. Decode | Setup | code | tool | explain | avg t/s | |---|---|---|---|---| | Stock Splash 1.4.0 | 31.6 | 29.5 | 25.4 | 28.8 | | + cache-aware routing (bias 2, top 4 protected) | 48.6 | 45.7 | 35.1 | 43.1 | | + desk-first overlap (current) | 48.9 | 57.3 | 40.3 | 48.8 | Draft acceptance: ~.70 code, ~.80 tool calls. Real agent use in pi: ~45 t/s average, 0 prompt-cache losses. Prefill (cold, no prompt cache) | Prompt | Time | pp t/s | |---|---|---| | 32,620 tok | 62.4 s | 522 | | 63,201 tok | 130.4 s | 485 | Splash keeps the prompt cache on disk, so in agent use only the new part of each turn gets read. What the fork changes (~800 lines, all behind env flags) Cache-aware routing leaves the router's top 4 experts alone and biases picks 5–10 towards experts already in memory. Halves SSD reads but changes output slightly (Quality check: on 20 LiveCodeBench problems stock scored 17 and the fork 16; agent and 32K/62K long-document tests were the same. One run each, so ±1 is noise.). Desk-first overlap starts the GPU on resident experts while the missing ones are still being read; output is token-for-token identical (11/11 on a fixed gate set). Lazy vision (still testing) loads the 864 MB vision tower only for image requests, so text keeps all 6,800 slots and identical output. Loading it the normal way cost 656 slots and 5.5% decode. Fable / Deepseek did 99.9% of this and as it’s a fork of Splash with only 800 new lines it’s 99.9% stock too. Also I get this isn't going to be helpful for many people, for me it's much much better than Tiel (at a quant I could run) and much faster than 27b so I'm happy :)