Ling-3.0 (BailingMoE3) lands in llama.cpp mainline - Quick benchmarks on Intel Arc B580

Finally llama.cpp now officially supports Ling-3.0! (Starting from build b10472 +) If you want to run them locally, bartowski has already released the GGUF imatrix quantizations for both models: - Ling-3.0-tiny (8B) - Ling-3.0-flash (127B) After quite a while, PR #26608 has officially been merged into master! There are still a few minor details left to iron out, but BailingMoE3 support is now fully official in llama.cpp. For my B580 on Linux I used: ./llama-bench -m Ling-3.0-tiny-Q8_0.gguf -ngl 99 -fa on --cache-type-k q8_0 --cache-type-v q8_0 -p 16384 -n 128 -n 32 -r 1 And these were the results: | model | size | params | backend | ngl | type_k | type_v | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.83 GiB | 7.89 B | Vulkan | 99 | q8_0 | q8_0 | 1 | pp16384 | 120.76 ± 0.00 | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.83 GiB | 7.89 B | Vulkan | 99 | q8_0 | q8_0 | 1 | tg128 | 114.24 ± 0.00 | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.83 GiB | 7.89 B | Vulkan | 99 | q8_0 | q8_0 | 1 | tg32 | 114.78 ± 0.00 | build: 9d77fa172 (10488) I also ran a test with a larger context: ./llama-bench -m Ling-3.0-tiny-Q8_0.gguf -ngl 99 -fa on --cache-type-k q8_0 --cache-type-v q8_0 -p 32768 -n 32 -r 1 Plaintext | model | size | params | backend | ngl | type_k | type_v | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.83 GiB | 7.89 B | Vulkan | 99 | q8_0 | q8_0 | 1 | pp32768 | 62.53 ± 0.00 | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.83 GiB | 7.89 B | Vulkan | 99 | q8_0 | q8_0 | 1 | tg32 | 110.49 ± 0.00 | build: 9d77fa172 (10488) Overall, the speed is quite good. I can actually run the full 128K context within my 12GB of VRAM using -c 131072 -ngl 99 -fa 1 --cache-type-k q8_0 --cache-type-v q8_0 . While I still need to test if it can be pushed even further since a few warnings pop up, it is entirely feasible and remarkably fast for a local setup.