Efficient Scheduling Of High Performance Batch Computing For Analytics Workloads With Volcano
Talkat KubeCon + CloudNativeCon North America 2022 · on Wednesday, 2022-10-26· at 14:30· Portside Ballroom (Room 260)·
summary
ING runs a self‑service analytics platform serving over 300 internal projects and more than 2000 users and replaces Hadoop/YARN by adding Volcano to Kubernetes plus an extension to Apache Spark binaries so Spark interactive sessions in Jupyter use Volcano queues. Volcano provides gang scheduling, preemption and hierarchical dominant resource fairness so dynamic allocation moves executors between users (demo shows executors auto‑rebalanced to 34→17→12 as users join), dashboards and automatic queue management are built and operate in production.
AI-written summary · about
chapters (10)
- 0:00Data analytics platform intro
- 4:17Self-service cloud-native platform
- 6:42Job scheduling in Kubernetes
- 9:52Volcano for Kubernetes scheduling
- 13:23Volcano scheduler features
- 19:30Spark with Volcano scheduler
- 21:39Volcano scheduler demo
- 24:51Workload monitoring and fairness
- 26:57Volcano contributions and future
- 27:41Conclusion and Q&A
related talks
43mRun Your AI Workloads and Microservices on Kubernetes More Easily and Efficiently...
26mIntro: Kubernetes SIG
30mVolcano: A Unified Scheduling Platform for Cloud Native AI
32mVolcano: Intro & Deep Dive
08mNext-Gen AI Orchestration With Volcano On Kubernetes
29mData Processing Efficiency: Optimizing Batch Workloads on Kubernetes With Custom Schedulers
37mImproving Performance of Deep Learning Workloads With Volcano
24mVolcano – Cloud Native Batch System for AI, BigData and HPC