How Much VRAM Do You Actually Need to Run a 70B Model Locally?

A single 24GB GPU can't run a 70B model — you need ~40-48GB. Here's the simple quant math to size any model, plus the three real ways to get there: stack GPUs, a unified-memory box, or rent.
评论
?
参与讨论