New Pattern for Sailing Multi-host LLM Inference
Talkat KubeCon + CloudNativeCon China 2025 · on Tuesday, 2025-06-10· at 14:30· Level 21 | Pearl Pavilion·
summary
Describes LeaderWorkerSet (LWS), a Kubernetes project for multi-host model-parallel LLM inference that targets cases where large models (e.g., Llama 3.1-405B, DeepSeek R1) cannot fit on a single node. Covers LWS features — dual Pod templates, fine-grained rolling updates, topology management, and all-or-nothing failure handling — and demonstrates integrations with inference engines like vLLM and SGLang plus adopter experiences from NVIDIA and Google.
AI-written summary · about
related talks
06mProject Lightning Talk: Sailing Multi-Host Inference with LWS
39mBeyond Model Sharding: Atomic Scheduling and Disaggregated LLM Serving With ...
33mllm-d: Kubernetes Native Distributed Inferencing
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
17mWaferLLM: Large Language Model Inference at Wafer Scale
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d