[Splash Engine] Qwen3.8-27B in native 8-bit at 37–55 tok/s on Apple Silicon: Extending Splash to Q8, 256k context scaling, and the "Reasoning Cliff"

https://preview.redd.it/nulsv53o8vqh1.png?width=4500&format=png&auto=webp&s=74765dbd409f4c221640f9f6000a685f6fdbb242

Spent weekend benchmarking the Splash engine (by Incoai) and extending its architecture to native 8-bit on Apple Silicon (M5 Pro, 64 GB unified memory).

Splash is a compiled C++ and Metal speculative decoding engine designed specifically for Apple Silicon. Upstream Splash pioneered a blisteringly fast speculative decoding pipeline for 4-bit models (~60 tok/s). However, aggressive 4-bit quantization hits a nasty "reasoning cliff" on competition-grade math and multi-step derivations.

We wanted to bring Splash's speed to true uncompressed 8-bit weights without losing its speculative decoding advantages. By extending Splash's architecture to support native 8-bit tiled Metal kernels (schema 5, MDFL0008), we were able to sustain 37–55 tok/s with zero quantization degradation.

• Model weights (27 GB native 8-bit): https://huggingface.co/nitinpanj/Qwen3.8-27B-Splash-HQ
• Q8 Metal runtime fork (Splash extension): https://github.com/npanj/splash/tree/q8
• Benchmark suite & raw JSON logs: https://github.com/npanj/splash-plus
• Benchmark & Context Scaling Plot: https://raw.githubusercontent.com/npanj/splash-plus/main/benchmark_and_context_scaling.png

Note on compatibility: Official upstream Splash 1.0 (incoai/splash) hardcodes package validation to 4-bit schemas (splash-packed-q4, schema 3/4). This fork adds schema 5 (splash-packed-q8, MDFL0008) loading and compiled Metal Q8 tiled decode kernels, while keeping 100% backwards compatibility with upstream Splash's official Q4 models.

Speeds on Apple Silicon (M5 Pro, 64 GB Unified Memory)

Evaluated at temperature=0.0 across 5 standardized task domains:

A few notes on the comparisons:

• Splash-HQ vs MTPLX (+39% overall, +92% math): Both run on the exact same 8-bit base weights. But Splash’s compiled C++ Metal backend executes with significantly lower dispatch overhead than Python/MLX DraftCore, getting 36.9 vs 26.5 tok/s overall, and hitting 54.8 tok/s on structured math reasoning.
• The Precision-Speed Paradox: Uncompressed native 8-bit (Splash-HQ, 27 GB) actually ran slightly faster on average than compressed 8-bit (Splash-Q8, 17 GB)—36.9 vs 36.5 tok/s. In speculative decoding, decode speed is $\text{Draft Speed} \times \text{Acceptance Rate}$. Aggressive compression flattened logits and lowered draft acceptance; native 8-bit produced sharper logits, fewer verification rollbacks, and higher net throughput despite reading more bytes from memory.

Context Scaling: What Happens Up to 256k Context (Live Telemetry to 190k)

Qwen3.8 is architecturally specified with a native 256k context window (262,144 tokens). Most Transformers fall off a cliff in decode speed as context grows because the KV cache balloons.

However, Qwen3.8 uses a hybrid architecture: 48 recurrent linear DeltaNet layers (fixed $128 \times 128$ hidden state, $O(1)$ memory growth with context) and only 16 full-attention layers.

On a 64 GB Mac, we pushed it live in an active server session all the way out to 190,016 tokens to see if decode speed degraded under real usage:

(See the visual plot in the repo: benchmark_and_context_scaling.png showing the full 51-point scatter and rolling trend line).

The big takeaway on context: Decode speed does not collapse. Thanks to Splash's memory handling and the hybrid architecture, it stays between 21 – 33 tok/s across the entire range.

The actual bottleneck at 150k+ context is cold prefill (TTFT). When the prefix cache hits, TTFT at 187k context is just 6.5 seconds. But on a cold cache miss, prefilling 180k+ tokens on a 27B model on Apple Silicon takes ~4–5 minutes. If you are using agent harnesses (like Oh My Pi, Claude Code, or curl), make sure client SSE idle timeouts are set high enough so the client doesn't drop the connection during cold prefills.

The "Reasoning Cliff" on Competition Math

Throughput numbers don't matter if math derivations hallucinate. We tested extended CoT reasoning on MATH-500, AIME 2025, and GPQA Diamond:

• On MATH-500 Problem 0 (evaluating $\sum_{j=1}\infty) \sum_{k=1}\infty) \frac{1}{(j+k)3} = p - q$), both stock Splash-Q4 and compressed Splash-Q8 fell off a cliff: they suffered numerical drift halfway through the algebraic series manipulation and output wrong values.
• Upgrading to Splash-HQ (full uncompressed 8-bit across all 64 layers) or Splash-Mixed (where only the top 8 sensitive layers, 56–63, are 8-bit) completely eliminated the cliff and cleanly derived $p - q$.
• Upgrading just the deepest 8 layers restored the full symbolic precision while keeping RAM manageable.

https://preview.redd.it/29mgrn0c9vqh1.png?width=2400&format=png&auto=webp&s=47bd228ef9096349e1c563a7d3ce0cc69743c505

Setup Recipe (How to Run & Download the Model)

The GitHub repo holds the C++ and Metal runtime engine, while the 27 GB model weights are hosted on Hugging Face. You don't need to manually download model files with git-lfs or separate scripts—Splash has a built-in package downloader.

1. Clone the Q8 runtime engine and build the Metal shaders

git clone https://github.com/npanj/splash.git -b q8 cd splash make -j4 

2. Launch the server (Automatic Download on First Run)

When you run the command below, Splash automatically detects missing model artifacts, connects to Hugging Face, streams the 27 GB files with progress bars, verifies the manifest SHA-256 hashes, and boots the engine:

./splash serve --model nitinpanj/Qwen3.8-27B-Splash-HQ 

(Once downloaded, subsequent runs load instantly from local disk offline).

(Optional: If you prefer to pre-download the model files beforehand via Hugging Face CLI instead, you can run:)

huggingface-cli download nitinpanj/Qwen3.8-27B-Splash-HQ --local-dir install/models/nitinpanj/Qwen3.8-27B-Splash-HQ 

3. Connect your client

The server exposes a standard OpenAI-compatible /v1/chat/completions endpoint on http://127.0.0.1:8000:

# Test via curl curl http://127.0.0.1:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "nitinpanj/Qwen3.8-27B-Splash-HQ", "messages": [{"role": "user", "content": "Explain why uncompressed 8-bit weights improve speculative decoding acceptance."}], "temperature": 0.0 }' # Or connect Oh My Pi (OMP) omp --model splash/nitinpanj/Qwen3.8-27B-Splash-HQ 

Practical Gotchas & Details

1. Memory headroom at 150k+ context: On a 64 GB Mac, model weights take ~27 GB. As context pushes towards 180k–190k, working memory climbs to ~42 GiB. Metal's memory governor will pause allocation growth when system free RAM dips below ~50 MB (Memory: growth paused). If you don't need 190k context, you can pass --max-context 131072 to cap it cleanly.
2. Backwards compatibility: You don't need two binaries. This fork preserves all upstream Splash 4-bit dense and MoE schemas (splash-packed-q4, splash-packed-q4-moe), so you can serve official models like incoai/Qwen3.8-27B-Splash or incoai/Qwen3.6-35B-A3B-Splash directly.
3. Only tested on Apple Silicon (Unified Memory): Everything here relies on unified memory bandwidth and Metal tiled shaders; not tested on CUDA or CPU.

Credits & Attribution

Full credit to the Incoai team for creating Splash (https://github.com/incoai/splash). Their C++ Metal speculative decoding architecture is what makes these speeds possible on Apple Silicon in the first place—this fork simply extends their work to support native 8-bit weights and custom Q8 tiled kernels. Also huge credit to the Qwen team for base weights and MTP architecture, and Youssofal for MTPLX reference benchmarks.

submitted by /u/SnooPredictions515