Qwen FN vs 27B --- Think I'm saturated.
Got QFN up and running on our Strix box this past weekend and have been running on it for a few days now. Big thank you/shout out to the Halogen team, it's running fantastic on the Strix, this is clearly "the setup" right now for this hardware with this model, really impressive performance for such a large model (~30-40TPS generation, ~900-1000TPS prefill on real world use, not benchmarks, over the past few days)! However, that said, I kind of feel like 27B saturated my personal use cases. QFN is a great model, but I'm not really noticing much where I think "Wow, 27B would never get this and QFN just one shot it". They feel very similar in capabilities (IE, both are amazing!) and I kind of feel like I'm reaching the end of the runway for what I can realistically make use of in my day to day use cases, I'm just asking questions that really require more than 27B the majority of the time. It really feels more to me like a very similar level of smart, one that runs well with limited memory bandwidth (QFN) and one that runs well with limited GPU memory (27B). It's an interesting result, and I guess maybe I should have expected it because I was really struggling already to find things that I actually need to do day to day professionally that 27B couldn't do. When I escalate to the cloud now it's nearly always for either speed or context, rarely intelligence. Surprising result, at least to me, I was expecting to have my hair blown back, but I guess this is just yet another data point that "good enough is good enough".