Next Level Spark on Kubernetes with Apache YuniKorn (Incubating)
summary
Apache YuniKorn replaces the Kubernetes default scheduler for Spark workloads and is running in production since July 2020, deployed at scale (small minutes-long jobs to multi-hour, many-executor jobs) to solve deadlocks, rigid namespace quotas and lack of job semantics. It provides hierarchical queues, gang scheduling, elastic resource quotas and bin-packing-friendly policies, integrates with the Spark operator and exports Prometheus metrics to DataDog. The result is multi-tenant isolation, observable scheduling and measurable cost savings via better autoscaling.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 2:26What is Apache YuniKorn
- 8:10Challenges with Spark on Kubernetes
- 13:51How YuniKorn schedules Spark
- 18:47Why YuniKorn is essential
- 19:52Apple’s batch platform requirements
- 21:37Default scheduler vs YuniKorn
- 24:03YuniKorn in production at Apple
- 26:33Platform architecture overview
- 30:29Spark operator and YuniKorn integration
- 31:33Monitoring queue usage with DataDog
- 33:21Future plans and enhancements
related talks
30mApache YuniKorn A Kubernetes Scheduler Plugin for Batch Workloads
30mBeyond Experimental: Spark on Kubernetes
29mData Processing Efficiency: Optimizing Batch Workloads on Kubernetes With Custom Schedulers
38mEnergize Multi-Tenancy Flink on K8s with YuniKorn
40mApache Spark on Kubernetes: A Technical Deep Dive
21mSpark on Kubernetes: The Elastic Story
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
29mCloud-Native Apache Spark - Spark on Kubernetes Is Now Generally Available