Qwen3.8-Flash-Next protip tensor-read-lazy on requires load-mode mmap

I expected the Q4 to fit on my system with tensor-read-lazy on but I kept running out of memory. Turns out load-mode auto wasn't using mmap. But once I turned it on manually, I got it to work! llama-bench -m models/qwen/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf --device Vulkan2/Vulkan1/Vulkan0 --tensor-read-lazy on --load-mode mmap -ngl 49 --split-mode layer -ts 19/10/20 WARNING: radv is not a conformant Vulkan implementation, testing use only. WARNING: radv is not a conformant Vulkan implementation, testing use only. ggml_vulkan: Found 3 Vulkan devices: ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: dot2 | bf16: 1 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat ggml_vulkan: 1 = AMD Radeon RX 9070 XT (RADV GFX1201) (radv) | uma: 0 | fp16: dot2 | bf16: 1 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat ggml_vulkan: 2 = AMD Radeon 8060S Graphics (RADV STRIX_HALO) (radv) | uma: 1 | fp16: dot2 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat model size params backend ngl dev ts lm lazy_mode test t/s qwen4exp A3B Q4_K - Medium 103.68 GiB 176.94 B Vulkan 49 Vulkan2/Vulkan1/Vulkan0 19.00/10.00/20.00 mmap on pp512 279.25 ± 5.91 qwen4exp A3B Q4_K - Medium 103.68 GiB 176.94 B Vulkan 49 Vulkan2/Vulkan1/Vulkan0 19.00/10.00/20.00 mmap on tg128 26.63 ± 0.27 build: d7bd3bfca (10680)

添加评论
点赞收藏
点踩分享查看原文
评论
?
参与讨论