Prefill vs. decoding and local LLM ROI: is prefill underrated?
I'm trying to understand why, when people discuss the ROI of running LLMs locally, they almost always focus on output speed (decoding) and rarely on input speed (prefill), which seems like it could have a significant impact on hardware ROI. Yesterday I saw a post on X where someone was running GLM 5.2 on 4 NVIDIA DGX Spark (4bit, speculative decoding, and other optimizations), achieving around 60 output tokens/s with 6 concurrent users in batch. Those are already great numbers. Assuming a hypothetical 24/7
评论
?
参与讨论