Prefill vs. decoding and local LLM ROI: is prefill underrated?

I'm trying to understand why, when people discuss the ROI of running LLMs locally, they almost always focus on output speed (decoding) and rarely on input speed (prefill), which seems like it could have a significant impact on hardware ROI. Yesterday I saw a post on X where someone was running GLM 5.2 on 4 NVIDIA DGX Spark (4bit, speculative decoding, and other optimizations), achieving around 60 output tokens/s with 6 concurrent users in batch. Those are already great numbers. Assuming a hypothetical 24/7

添加评论
点赞收藏
点踩分享查看原文
评论
?
参与讨论