Flyweight: open-source C++/CUDA engine for running MoE models bigger than your VRAM on one GPU + system RAM. First PyPI release, looking for contributors.

Been building this for a few months, mostly for myself, and it just got a proper release so figured I'd post it. It's a native GGUF inference runtime with OpenAI/Anthropic-compatible APIs and a chat UI. The whole point is one consumer NVIDIA card + lots of RAM: MoE models that don't fit in VRAM run their experts on the CPU, or split with a hot set cached on the card. It figures out what fits at startup instead of you guessing offload layer counts. Runs Qwen 3.x dense and MoE (incl. Qwen3.8-Flash-Next), DeepSeek-V4-Flash, Ling 3.0, K2-Horizon, Gemma 4, Laguna, Muse Glimmer. Image input via mmproj on the Qwen models. Also does Z-Image-Turbo image gen next to a chat model on the same card. Numbers from my laptop (5070 Ti 12 GB, 60 GB RAM): - Qwen3.8-Flash-Next IQ1_S: ~35 tok/s decode, ~475 tok/s prefill - Qwen3.8-27B IQ2_XXS: ~40 tok/s - DeepSeek-V4-Flash: 6-7 tok/s (that's basically the DRAM bandwidth limit) - Z-Image 1024x1024 in ~15 s with a 35B loaded beside it Stuff I think is neat: - Kernels are compiled at runtime by NVRTC, so no CUDA toolkit in the wheel and no nvcc. Same kernel source compiles as plain C++ for a CPU-only backend. - KV cache in f16 / q8_0 / TurboQuant 4-bit. On the dense 27B at 32K that's the difference between 10.8 and 22.9 tok/s, because it's what keeps the weights on the card. - Tool calls are enforced by a sampler grammar, and temperature/penalties are clamped inside a call so edit tools reproduce file text exactly. Makes Claude Code / opencode a lot less flaky on small quants. - Thinking budget is a hard cap, plus a stop_thinking endpoint to cut a stream over to the answer. Full disclosure: a lot of this was written with AI help (Claude Code, mostly). I did the design, the measuring, and the arguing about what's actually faster; the AI did a lot of the typing. Every kernel is parity-tested against a reference and the perf numbers are real measurements, but if that's a dealbreaker for you, fair enough. pip install flyweight-llm flyweight doctor flyweight serve model.gguf Linux x86-64 and Windows wheels. Needs the NVIDIA driver + CUDA toolkit (for NVRTC and headers), or --backend cpu. github.com/yairpatch/flyweight (Apache-2.0) Where I could use help One-person project, one-person blind spots. PRs and issues welcome, especially: - Runs on other hardware. Everything was measured on one Blackwell laptop and one Intel box. Ampere/Ada, AMD CPUs, 8 GB cards, all untested. Even just flyweight doctor output + a tok/s number in an issue helps. - Windows users. CI passes, real usage is thin. - macOS / ARM. No wheels, nobody's tried. The CPU backend should build. - Models llama.cpp runs that this doesn't. Open an issue with the GGUF metadata. - CPU expert kernels, especially the 1-3 bit IQ formats. That's the bottleneck on everything MoE. - Anything you tripped over installing. There's a plans/ dir with notes on what's been tried and what got dropped, so check there before proposing something. CONTRIBUTING.md has the rest. Happy to answer questions.