JetBrains Mellum 2.1 12B A2.5B MoE is a very good model.

localllama (Reddit)，r/LocalLLaMA，约 72.8 万成员的社区，专注于在本地硬件上运行大语言模型，涵盖模型选择、GPU 配置、量化技术、Ollama 与 llama.cpp 等工具，以及隐私优先的 AI 工作流。关注

It runs at around m4 16 at 60 tps at full 130k context, you can easily run a Q8 if you have a 16 GB GPU, I was able to get 140TPS on 9070xt, It’s an MoE so I think we can run it well on smaller cards too, 8GB, 12GB should be possible to get upwards of 30-40 tps. I have used it a bit, it can code, but I think having a bigger model plan tasks in detail would make it quite good for quick task execution.

添加评论

点踩

分享查看原文

参与讨论