Compact Rollback MTP: a MTP version for QWEN models for those with little vRAM

I've made a modification of llama.cpp MTP for people that want to run models like QWEN 27B on 16GB and similar setup, the focus is reducing the memory cost of MTP allowing more speed for less ctx cost. MTP Mode Maximum Draft (n) Available Context TG (t/s) Standard 2 72,192 39.53 MTP Compact Rollback 5 77,312 46.39 On this example of a (well tuned!) IQ4 running on 16GB you get some +5k ctx and enjoy 17.35% increase on token generation. With MTP the more speculative tokens you generate (n-max) the more the speed increase chance you get yet you pay a price in vRAM for those (rollbacks in case the latest token are not accepted) and that reduces the ctx available. Now we use dense 27B and A3B mostly to generate code and you know that we have stellar acceptance rates: like 80-98%. MTP Compact Rollback allows the user to limit how many immediate MTP rollback states the model keeps in VRAM. For example, --spec-mtp-cr-depth 1 --spec-draft-n-max 5 keeps only one immediate rollback snapshot while still allowing MTP to generate five tokens, the user is not constrained to compromise on a lower MTP to preserve a decent ctx length, max MTP draft depth like 5 or 7 can be used for the same context cost of 1. The idea of keeping just one rollback plays nice with Adaptive MTP implemented in this patch: --spec-draft-adaptive Dynamically adjusts the MTP draft limit, up to --spec-draft-n-max , based on recent draft acceptance. Adaptive speculative decoding is useful when mixing tokens generations in different domains, es coding sessions that can benefit from an higher n-max, creative prose (that can happen in long reasoning traces in coding too) that uses lower n-max. This costs very little computation so always use it. How to dwl and apply: git clone github.com/ggml-org/llama.cpp cd llama.cpp git checkout 662a0b01 wget store.piffa.net/lm/bug/mtp_compact_rollback_662a0b01.patch git apply latest_rocm_improvement_662a0b01.patch Build llama.cp as usual (ask your LLM for help if never did that). Quick minimal implementation: add to your llama-server script: --spec-mtp-cr-depth 1 --spec-draft-adaptive --spec-type draft-mtp Recommended full settings for coding on 27B and A3B: --spec-mtp-cr-depth 1 --spec-draft-adaptive \ --spec-type draft-mtp,ngram-mod --spec-draft-p-min 0.80 --spec-draft-n-max 5 \ --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \ --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 8 --spec-ngram-mod-n-max 32 Example scripts: store.piffa.net/lm/bug/llama_scripts More info, docs, artifacts: store.piffa.net/lm/bug Previously tested on: