[Guide] Ternary Bonsai 2 is finally 100% working on ik_llama.cpp (Native CUDA offload + Fast CPU -rtr)

Hey folks, Quick heads up for anyone playing around with PrismML's Ternary Bonsai 2! Until very recently, running it outside of Prism's own fork was a bit of a pain. Upstream llama.cpp uses a block size of QK=256 (TQ2_0), while Prism uses QK=128 (PQ2_0), and trying to offload layers to CUDA on other forks would bottleneck or break due to Hadamard sign tensor handling. With the latest commits landed in ik_llama.cpp, we finally have complete, native support across both CUDA and CPU. Here is how to set it up cleanly. Convert upstream weights losslessly (PR #2694) If you already have a native PQ2_0 GGUF downloaded from Prism, you can skip this. If you downloaded upstream TQ2_0 or TQ1_0 files, ik_llama.cpp now ships with a script that losslessly repacks them to PQ2_0 in just a few seconds (exact trits preserved, no requantization needed): python3 scripts/repack_ternary_to_prism.py Bonsai-2-27B-TQ2_0.gguf Bonsai-2-27B-PQ2_0.gguf Native CUDA Full Offload (PR #2691 & PR #2747) Native CUDA kernels for PQ2_0 are now merged. You can offload everything to your Nvidia GPU without token corruption or falling back to heavy k-quants: ./build/bin/llama-cli -m Bonsai-2-27B-PQ2_0.gguf -ngl 99 -p "Explain the theory of relativity in simple terms." --temp 0.7 --top-p 0.95 --top-k 20 CPU Inference with fast prompt ingestion (-rtr 1) If you're testing on CPU, ik_llama.cpp uses an 8-row interleaved format (PQ2_0_R8) backed by iqk_mul_mat kernels. Remember to pass -rtr 1 (runtime repack) to drastically accelerate prompt processing: ./build/bin/llama-cli -m Bonsai-2-27B-PQ2_0.gguf -rtr 1 -t 8 -p "Explain the theory of relativity in simple terms." --temp 0.7 --top-p 0.95 --top-k 20 TL;DR: No more sign tensor bottlenecks on GPU. PR #2747 brings native CUDA kernels, PR #2694 gives us instant lossless repack from upstream weights, and CPU prompt ingestion with -rtr 1 is flying. Big shoutout to Iwan Kawrakow and the contributors for getting ternary models working natively!