Why is MTP slower than standard decode?
I'm trying to get Ornith 1.5 9B running (I can also barely run 35B A3B but I want something that I can run relatively easily) with the latest llama.cpp with this command: llama serve --flash-attn on --cache_type-k q8_0 --cache-type-v q8_0 --hf-repo ornith-ai/Ornith-1.5-9B-GGUF --spec-draft-p-min 0.9 --spec-type draft-mtp And this gives me ~15 tok/s TG and ~200 tok/s PP. However, without MTP, I am getting double these numbers. I have a very high draft acceptance rate of 0.95 with a mean length of 2.27. I have an 8 GB RTX 4060 by the way, but I feel this is irrelevant. I have noticed with MTP my memory bandwidth and GPU utilisation is about half of what it is without MTP.
评论
?
参与讨论