35B MoE running at 20.5 tok/s on an iPhone, without skipping experts

I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting. Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation fast enough. OpenBMB’s MiniCPM-V 4.7 uses a technique called Lookahead Expert Scheduling (LES) to run a 35B-A3B MoE model locally without changing its actual routing or dropping experts. The key idea: predict which experts the next layer might need, preload their weights from flash storage while current computations run, and execute ready experts first. Combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, LES achieves: 20.51 tok/s on iPhone 18 Pro Max 6.07× faster than the 3.38 tok/s baseline 26.7% smaller expert weights with only a 0.16-point drop across five benchmarks Around 3 GB peak memory in a low-memory configuration The implementation, including expert scheduling and DSpark integration, is open source in llama.cpp-omni . The interesting part is that LES hides memory bottlenecks without changing which experts the model actually executes. This approach could potentially work with other MoE models, too.