Local one-shots, you say? Yep, we're there with Qwen 3.8 27B

ootsby:

This is silly. The average user is buying…

ootsby:

Okay, but that’s also the gap between realistic local hardware for even enthusiasts and esoteric builds or spending car money for hardware that doesn’t stand up…

“realistic local hardware for even enthusiasts”

Hence talking about 32GB cards, and the fact that your assertion that a neutered model with insufficient context for the job of working on complex codebases running on a 16GB card with frontier models is not relevant to the majority of people running local LLMs. Remember? That’s where we started - you trying to say that we’re not there yet because of your personal experience.

It’s pretty simple. I’m not suggesting that everybody buy a 170HX, I’m saying that in the consumer/workstation class (which includes dual 3060s all the way up to R9700s) there are a whole ton of people putting at least 32GB VRAM to use for this, for the cost of a single 5070 Ti or less - which is apparently your high water mark.

That includes a significant portion of this forum’s active users. Unless you think that the 6 million people who’ve downloaded Unsloth’s 3.8 27B model alone were all doing so with the 2-bit and 3-bit quants?

ootsby:

The average user is…

Nobody’s talking about the average user here. The average user doesn’t even have a discrete GPU in their system.

Unless, of course, you want to talk about Mac users who’ve bought a 64GB laptop in the last couple of years. They can run Q4-Q8 quants too, and last I checked there were quite a lot of people in that position.

ootsby:

The card you’re talking about isn’t even available now for the prices you’re talking about. Good for you for doing the homework and taking the risk and all that but it’s really not an option for anyone.

I bought it about two weeks ago. There are plenty of them on eBay for £950-1050 and accepting offers, which is exactly what I did.

The problem here is your extrapolation. Just because it’s not an option for you, doesn’t mean it’s not an option for anyone. If somebody has a need for 64GB VRAM, even if the price of these goes to £1500, it’s still the cheapest way to do it. Perhaps not the best, but definitely the cheapest.