2.5x faster Qwen3.6 NVFP4 Unsloth quants

localllama (Reddit)，r/LocalLLaMA，约 72.8 万成员的社区，专注于在本地硬件上运行大语言模型，涵盖模型选择、GPU 配置、量化技术、Ollama 与 llama.cpp 等工具，以及隐私优先的 AI 工作流。关注

Hey r/LocalLLaMA folks! We made NVFP4 quants 2.5x faster for Qwen3.6 27B and also 1.56x to 1.79x faster for 35B-A3B vs NVIDIA's NVFP4 quants without any accuracy degradation! We used W4A4 so actual 4bit tensor cores for matmuls, whilst NVIDIA's ones uses W4A16. FP8 KV Cache calibration is also provided, auto allowing 2x longer contexts . For accuracy we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s. It also has MTP pre-embedded. We also provided 2 35B versions NVFP4-Fast (

添加评论
点赞收藏
点踩分享查看原文
评论
?
参与讨论