llm-d 1.0 adds production controls for distributed AI inference
Carlos Costa spent years working on large-scale supercomputers before helping create llm-d to bring cloud-native methods to the problem of serving AI models at scale. On October 20th, the project reached its 1.0 release, adding operational controls for running distributed inference after deployment, including multi-tenant traffic priorities, router failover and rolling upgrades.
评论
?
参与讨论