Optimizing Training Performance for Large Language Model(LLM) in Kubernetes
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 17:00· Level 1 | Hall Entrance S10 | Room A·
summary
Volcano adds a HyperNode CRD and topology-aware scheduling (hard/soft modes) to confine LLM jobs to rack/row boundaries, tested in a simulated cluster using kind and quark; 18-node racks with 4 GPUs/node yield 72 GPUs per rack, jobs of 18 nodes (72 GPUs) and 54 nodes (216 GPUs) consistently land inside one rack or one row in the demo. The feature runs under Volcano v1.01.01 preview, works in simulation, and still needs automatic network discovery from cloud providers and robustness for dynamic node changes.
AI-written summary · about
chapters (11)
- 0:00Optimizing LLM training
- 3:51Volcano project overview
- 7:08LLM training challenges
- 8:11Networking-aware scheduling
- 12:13Demo motivation
- 16:35Cluster topology view
- 18:06Rack-level scheduling demo
- 21:50Row-level scheduling demo
- 24:07Enabling network topology
- 26:56Current approach challenges
- 28:24Future directions
related talks
26mIntro: Kubernetes SIG
37mImproving Performance of Deep Learning Workloads With Volcano
32mVolcano: Intro & Deep Dive
30mVolcano: A Unified Scheduling Platform for Cloud Native AI
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
22mIntroduction to Large Language Models (LLM) on Kubernetes
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine