Intro To Volcano: Cloud Native Batch System
Talkat KubeCon + CloudNativeCon North America 2022 · on Wednesday, 2022-10-26· at 17:25· Virtual Platform Only·
summary
Volcano extends Kubernetes to run batch and HPC workloads, now at incubation in CNCF with feature releases every 3 months; the project reports ~450 contributors and adoption by 50+ production companies. It provides multi-pod job APIs, scheduler plugins, a cluster-scoped Q for weighted resource sharing, fair-share and reservation policies, elastic training (auto-scale replicas to improve GPU utilization and lower cost), cross-region scheduling and Spark 3.3 integration. Users report higher scheduler throughput for massive video transcoding and cost reductions via collocation and oversubscription.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:42Adopters
- 2:30Volcano architecture
- 4:25Scenario: internet audio/video transcoding
- 6:48Scenario: Genetic/drug discovery/biopharmacine
- 9:15Scenario: Traditional HPC migrate to cloud native
- 11:01Job management
- 13:24Resource management: Queue
- 14:13Dynamic sharing between queues
- 15:45Fair share within Queue
- 16:51Scheduling policies in Volcano
- 17:19Elastic scheduling
- 19:34SLA scheduling
- 21:42Batch scheduling for Spark
- 22:44Release
- 23:41Resources
related talks
32mVolcano: Intro & Deep Dive
24mVolcano – Cloud Native Batch System for AI, BigData and HPC
26mIntro: Kubernetes SIG
43mRun Your AI Workloads and Microservices on Kubernetes More Easily and Efficiently...
30mVolcano: A Unified Scheduling Platform for Cloud Native AI
30mCloud Native AI: Harness the Power of Advanced Scheduling for High-Performance AI/ML Training
37mImproving Performance of Deep Learning Workloads With Volcano
08mNext-Gen AI Orchestration With Volcano On Kubernetes