llm-d 1.0 adds production controls for distributed AI inference

Carlos Costa spent years working on large-scale supercomputers before helping create llm-d to bring cloud-native methods to the problem of serving AI models at scale. On October 20th, the project reached its 1.0 release, adding operational controls for running distributed inference after deployment, including multi-tenant traffic priorities, router failover and rolling upgrades.

添加评论
点赞收藏
点踩分享查看原文
评论
?
参与讨论