Gewell - Gemma4 inference engine

the What An engine to run Gemma 4 31B on blackwell under massive concurrency and rather specific workload patterns. I've been waiting for someone to do ninfer but for gemma, and, well, ended up having to do it myself. More models and potentially more gpus are likely to be added, but its main purpose is to be my own workhorse, and I do not have the capacity (or desire) to chase every new release. I do love the gemma 4 family as a whole tho, so they are very likely coming soon. # the Why Ironically, there has just been a post on "stop making slop inference engines", so... why bother with own engine if vllm exists? Well, neither vllm nor lcpp dont utilize one of the Gemma's big strengths, which is being able to have your kv cache use 0.625x the vram losslessly. Not "trust me bro" losslessly, but like, mathematically losslessly down to the order of reduction. Why? Because they decided to tie K and V weights on global attention layers, and rope only rotates 25% of K. So we can store only V and 25% of K, while other engines store full K and V. It is slightly more computationally intensive to have to unsqueeze them for the math, but it very quickly becomes outweighed by having to read less from memory. Blackwell has way more compute than vram bandwidth. And, well, lets you pack more context or more cached prefixes into the same amount of memory. Also, vllm's cache sucks. Like, really sucks. It is good for when you have a lot of random users sending random prompts, but lack of explicit cache controls and LRU policy really makes some loads suffer, and SWA snapshots are clearly an afterthought (cant blame them for that because vllm predates SWA by a few years, but still). Gewell is built around efficient use of checkpoints, ram offloading and both smarther default eviction policy that assumes you are going to have repeating prompts with significant intervals and explicit cache hints on the prompts themselves. More about how cache works here: github.com/leDissolution/gewell/blob/main/docs/cache.md Tl;dr: say, you have two chats going on you are alternating between. If you send ten messages into one of them in a row, vllm will make 10 checkpoints and evict the otehr one; gewell will dissolve some of the the intemediate checkpoints and preserve the second one warm. Why it is important? Well, I'm using gemma for data generation and grooming, and most of these workflows have writer + ctitic or planner + writer + critic loops, sometimes with even more separate prompts cycling around. Each of these prompts is building up on top of its own's previous turn history so their prefixes are perfectly reusable, but vllm insists on pushing them out. It gets even worse if there are some one-off prompts that arrive every 10-20 turns and will never be reused, yet they still take up prefix cache and evict something useful. Gewell also starts fast. Like, fast. Literally couple of seconds on top of reading the weights from the drive, because instead of doing live kernel profiling to select gemm shapes the choices were profiled offline and hardcoded and there is no python import tax. # the How Fast Decently fast. TTFT is generally slightly behind vllm on large batches (because scheduler prioritized saturating decode width over latency and high-batch prefill is slightly slower for lower quants), but overall t/s is generally higher - especially on the workload it was designed for (bunch of prompts that keep growing but not all active at the same time). preview.redd.it/tzn2iprbfuqh1.png preview.redd.it/cho1porbfuqh1.png preview.redd.it/2ul22prbfuqh1.png # the Quants Gewell uses its own quant format that allows for arbitrarily mixed precision. The convertion tool lets you repack any compatible checkpoint with whatever bpw you want. The "main" quant it was developed around is G0: huggingface.co/LeDissolution/Gemma-4-31B-it-Gewell_G0 It uses around 6bpw, allocating most of them into attention and global-attention-adjacent MLP. Why not qat? Well, because it is kinda bad in my experience (especially in the context fidelity and vision). Nvidia's nvfp4 was my go-to, but my personal tests showed that 16bit in attention are mostly wasted and mlp needs some juice too. Intuition being that if we take the beautiful precise 16-bit attention and then pass it through 4-bit up-gate, we just lose all that fine detail anyway. Idk whether it is mechanically correct, but seems to work? YMMW. preview.redd.it/3e02slcefuqh1.png preview.redd.it/6phv3wcefuqh1.png preview.redd.it/gayesosvfuqh1.png The tasks here are ~2.5k example mix pulled from aya_dataset, OpenR1-Math, DocVQA, ChartQA, QASPER and code_contests NIAH is a RULER-inspired torture test where the model is fed a huge uniform block of key-value pairs with distractors and overwrites: Record 3832768 stores value ocean. ... Record 3832760 stores value rose. Record 3832761 stores value pearl. ... Record 3832767 stores value ocean. Record 3832768 stores value river. Requested keys in order: 3832768 3832760 .... And the model needs to respond with exactly the same amount of values in the exact requested order. Amount of needles is 16 for the current test set; completion was counted as % of the correct values in correct spots. At 64k even bf16 can not complete a single request perfectly without reasoning. # the Supported Hardware It was developed and tested on linux and pro 6000. I have not tested it on 5090 because I dont have it, but the intent behind choosing the quant size was to have the weights + mtp + 250k context fit in 32gb. Adding vision might require reducing the context size a bit. Windows support was not tested either (my windows machine got 3090s), but there is nothing that prevents it in principle, so you are welcome to try. # the Limitations I did cut some corners on the interfacing side. The samplers support is currently very rudimentary (only temp, top-k and top-p), there is no way to override the chat template (the latest google's one is hardcoded in), and some less common text/chat completion knobs might be missing. # the Roadmap There are likely some bugs to be fixed I did not find when using it myself, and some more works has to be done around the API. Next big thing I plan is supporting 26A4, but no promices when. I also have a bunch of ideas around better speculative drafting, and it might or might not come before 26A4.