More Than Model Sharding: LWS & Distributed Inference
Talkat KubeCon + CloudNativeCon China 2025 · on Tuesday, 2025-06-10· at 14:30· Level 19 | Crystal Court I·
summary
Covers distributed inference for very large LLMs (e.g., Llama3.1-405B, Deepseek-V3 671B) running across multiple nodes with backends like vLLM + Ray, and the operational problems that arise on native Kubernetes (uncoordinated StatefulSets, gang-scheduling needs, startup ordering, HPA scaling, stable indexing/ranks, topology-aware grouping, and failure recovery). Presents LeaderWorkerSet (LWS, github.com/kubernetes-sigs/lws) as a Kubernetes primitive to coordinate leader/worker groups: enabling co-location, group-aware HPA, gang-style scheduling, and an all-or-nothing restart policy to improve performance and fault tolerance for distributed inference workloads.
AI-written summary · about
related talks
33mllm-d: Kubernetes Native Distributed Inferencing
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
38mOptimizing LLM Inference for the Rest of Us
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes
34mImprove AI Inference (serving models) With KServe and VLLM
31mFrom Model Serving To Distributed Inference: How Llm-d Evolves AI Platforms on…