Here my pretty good qwen3.8 27B setup, hope it helps

Since I spent the time to figure it out and it is not like it will make me any money ever. I think I'd share with you all what I managed to cobble together.

First and foremost here the relevant bits about my hardware and software stack:

OS: Debian 13
CPU: 8700G
GPU: 7900XTX (this GPU has 24GB VRAM as all the others of its kind)

I definitively have system ram but this configuration does not touch it very much so I won't go over that.

A bit of history, I tried something alike with the previous Qwen3.6-27B but I can't really explain why it really crawled. I may blame the MTP in part and this setup seems to solve that.

so...
I chose for the quant unsloth's Qwen3.8-27B-UD-IQ4_XS.gguf to leave some room. and that without any fancy setup reached around 22 to 30 t/s which is pretty good.

Note that the UD prefix seems to be doing something significant I can't quite explain and it is recent-er than the launch quants unsloth offers so consider a redownload as it is smaller than the non UD counterpart at the very least.

I was deeply perplexed about my previous failure at MTP since many people spoke about how much faster it made their model, and it did. until I got to long chats. so I sent chatgpt to the documentation mines because honestly I don't read the llama-server docs before going to bed and it came up with option --spec-draft-p-min which seems to do quite a bit of work here as it rejects further MTP immediately as it sees low confidence (than the treshold) I tried to move it around but the number I settled on seems the perfect one.

Then lastly that is interesting and here I will then cut it out, I tried the kv cache quantization again, now ti doesn't freak out while I think it previously did. lastly really in short the 3 long MTP seems to be working fine, you may tune it either way if your workloads are different. I run a "benchmark" by making it build a vue UI thingy so my sort of generation is pretty boring and common and may not be representative of all the workloads.

so to end it all here the llama-server command I use, inside of llama-swap:

macros: models_dir: "${env.HOME}/.local/share/llama-swap/models" llamasrv: "${env.HOME}/.local/bin/llama-server" models: qwen3.8: cmd: > ${llamasrv} --port ${PORT} -c 140000 --parallel 1 --model ${models_dir}/qwen3.8/Qwen3.8-27B-UD-IQ4_XS.gguf --mmproj ${models_dir}/qwen3.8/mmproj-F16.gguf --model-draft ${models_dir}/qwen3.8/mtp-Qwen3.8-27B-Q4_0.gguf --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.70 --spec-draft-ngl all --cache-type-k q8_0 --cache-type-v q8_0 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --flash-attn on -ngl 999 capabilities: context: 128000 

At runtime the model takes about all the VRAM around 22.1 GB, change the settings by reducing context size or if you want using a lower quant if you have some more "professional workload" for that VRAM and you might still be able to run some games or apps that are not your desktop or browser if you want, like blender or your new AI based videogame for some reason (I make software, this setup is for making software and i tend to not make video games that are very beautiful or good the few times I try, and I won't use enough of games or AI at the same time for me to matter).

and yes, there is a llama-swap setup. it is a pain to change it every time but it is less of a pain than doing more manual labor to change more stuff every time you update a model, also my llama swap is started by a systemd service. why not docker? docker is a huge bother for such close to the host management... like. I usually do use docker but all the permission mess with folders and due to how often I want to maintain my setup I'd rather not.

# /etc/systemd/system/llama-swap.service [Unit] Description=llama-swap service After=network-online.target Wants=network-online.target [Service] Type=simple User=gabrielesilinic Group=gabrielesilinic ExecStart=/home/gabrielesilinic/.local/bin/llama-swap -listen 0.0.0.0:42134 -config /home/gabrielesilinic/.local/share/llama-swap/config.yaml Restart=on-failure RestartSec=5 # Basic hardening NoNewPrivileges=true PrivateTmp=true ProtectSystem=full # Allow normal access to your home directory. # Tighten this later if desired. ProtectHome=false [Install] WantedBy=multi-user.target 

my systemd service is possibly wrong and weird but it works out. don't worry about it. I just didn't want to refactor it further. it is nonetheless fairly reliable.

and additionally my opencode config so far, which btw has playwright installed.

{ "$schema": "https://opencode.ai/config.json", "model": "llama-swap/qwen3.8", "subagent_depth": 0, "permission": { "websearch": "allow", "webfetch": "allow", "playwright_*": "allow" }, "mcp": { "playwright": { "type": "local", "command": [ "npx", "-y", "@playwright/mcp@latest", "--browser=chrome" ], "enabled": true, "timeout": 30000 } }, "provider": { "llama-swap": { "name": "llama-swap", "npm": "@ai-sdk/openai-compatible", "options": { "baseURL": "http://127.0.0.1:42134/v1" }, "models": { "qwen3.8": { "name": "Qwen 3.8", "attachment": true, "tool_call": true, "modalities": { "input": [ "text", "image" ], "output": [ "text" ] }, "limit": { "context": 128000, "output": 32768 }, "options": { "chat_template_kwargs": { "reasoning_effort": "medium" } }, "variants": { "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } }, "xhigh": { "chat_template_kwargs": { "reasoning_effort": "xhigh" } } } } } } } } 

btw I don't know if the variant reasoning_effort works. probably it doesn't. I havent seen too much of a difference personally.

oh, technically if you want if you have the same amount of VRAM you can halve the context by setting llama.cpp to parallel 2 and have some subagenting. but I tried and this card basically survives and that is it.
I advise that to be done on stronger cards or for users who have more patience as it slashes the performance of the simultaneous generations significantly, I never tested it after MTP optimizing further but I don't wanna and there is no reason to as enabling subagenting for a model with no consistent behaviour about it and no limiter is not a great idea.

lastly for comfort consider adding to your .bashrc export OPENCODE_ENABLE_EXA=1 which enables a third party somehow free somehow opencode integrated provider. it is otherwise fairly difficult to get that working. it technically might sell your data? I don't know. but it really doesn't get much out of your queries and is probably rate limited (I haven't had my agent use it enough to get that) in any case I did all of this first to stop paying github copilot which is basically overpriced at this point.

I added as backup my chatgpt subscription which is somewhat restrictive to use sol with when qwen has a breakdown which btw is not very often at all unlike the previous model that just shat itself at the first difficulty and sometimes just looped forever.

This setup on this machine produces about 28 to 40 t/s and seems particularly useful since the thinking on this model is a lot.

I am pretty sure that with some tweaking everyone even maybe people with half my memory will be able to at least get a 64k window at decent speeds while the 3090 users may flex on me. as you saw I have trust issues and while I can technically fit more context on I haven't dared tell the harness. I will think about it.

It seems pretty stable though. previous attempts had real issues keeping the right context size in check, it may be due to llama-swap and llama-cpp getting updated. btw do download the latest release of both as they evolved significantly. llama-swap gave me useful stats that solidified my decision in settling onto this setup specifically.

additional tip, my llama-server is not "real"

#!/usr/bin/env bash set -euo pipefail export PATH="/home/gabrielesilinic/.local/bin:/usr/local/bin:/usr/bin:/bin" export LD_LIBRARY_PATH="/home/gabrielesilinic/.local/opt/llama.cpp${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" export ROCR_VISIBLE_DEVICES=0 export HIP_VISIBLE_DEVICES=0 exec /home/gabrielesilinic/.local/opt/llama.cpp/llama-server "$@" 

it is stupid but it works out.

I wrote a bunch of this from memory so some values may be a little bit off. but it won't matter very much. if you have the same GPU as I it will work well for you. if you don't it may work better or you are going to have to tweak it anyway.

And that is all, have fun!

submitted by /u/gabrielesilinic