Data Processing Efficiency: Optimizing Batch Workloads on Kubernetes With Custom Schedulers

on YouTube ↗29m

" summary

Built an open-source city cycling navigation platform and runs Apache Spark on Kubernetes across multi-cloud clusters using Volcano and Apache YuniKorn as custom schedulers. Tests simulate ~100 events/hour and up to ~1,000 jobs, plus node failures via spot instance kills, and show 2–3x cluster utilization versus the default scheduler. Tradeoffs: Volcano is feature-rich but complex to configure, YuniKorn is simpler with a clearer UI, both suffer occasional preemption-driven Spark driver deaths and middling scheduling latency that affect some interactive and time-sensitive jobs.

AI-written summary · about

" chapters (8)

1. 0:00Introduction and context
2. 2:01Batch workloads on Kubernetes
3. 7:53Default scheduler limitations
4. 11:40Historical context: kube-batch
5. 12:46Volcano custom scheduler
6. 16:35Apache YuniKorn scheduler
7. 18:51Benchmarking and performance
8. 28:19Conclusion

" related talks

• 30mBeyond Experimental: Spark on KubernetesWeiwei Yang · Kubernetes Batch + HPC Day North America 2022
• 36mNext Level Spark on Kubernetes with Apache YuniKorn (Incubating)Chaoran Yu, Weiwei Yang · ApacheCon@Home 2021
• 28mEfficient Scheduling Of High Performance Batch Computing For Analytics Workloads With VolcanoKrzysztof Adamski, Tinco Boekestijn · KubeCon + CloudNativeCon North America 2022
• 26mIntro: Kubernetes SIGZefeng Wang, Da Ma · KubeCon + CloudNativeCon Europe 2019
• 24mVolcano – Cloud Native Batch System for AI, BigData and HPCWilliam (LeiBo) Wang · Kubernetes Batch + HPC Day Europe 2022
• 43mRun Your AI Workloads and Microservices on Kubernetes More Easily and Efficiently...William Wang · KubeCon + CloudNativeCon + Open Source Summit China 2023
• 32mVolcano: Intro & Deep DiveKlaus Ma · KubeCon + CloudNativeCon Europe 2022
• 26mOptimal Approaches for Real-Time Machine Learning with Apache Spark on Kubernetes.Hichem Kenniche · Community Over Code EU 2024