Apache Yunikorn: State of Union and Future
summary
Apache YuniKorn is a stateless core plus shim scheduler deployed on Kubernetes with an admission controller and optional coexistence with the default scheduler; it claims to cut default scheduler tail latency (reported ~100s) for batch jobs, supports application-aware scheduling and hierarchical queues with min/max quotas, performs resource reservation via placeholders and node-level tentative allocations, and implements gang scheduling with task groups and placeholder-to-real pod swap. Releases 0.9–0.11 add gang support, Spark Operator integration and stability; a 2,000-node simulation shows near-linear fair sharing.
AI-written summary · about
chapters (19)
- 0:00Start
- 2:09Agenda
- 5:05Gaps in the Scheduling
- 6:06Challenges
- 7:24Introduction to the Unicorn Scheduler
- 9:55High Level Architecture of the Unicorn
- 10:34Unicorn Core
- 10:57Shim Layer
- 12:27App Aware Scheduling
- 13:49Quota Enforcement
- 17:02Resource Fairness
- 18:18Resource Reservation and the Spark Operator Integration
- 18:56Spark Operator Integration
- 19:32Gang Scheduling
- 20:17Task Groups
- 20:41Prepare the Application
- 23:19Present and the Future of the Unicorn
- 34:00Does Unicorn Interface with Kubernetes through a Plugin or Do You Need a Forked Version of Kubernetes
- 38:23Benchmark Results
related talks
30mApache YuniKorn A Kubernetes Scheduler Plugin for Batch Workloads
38mEnergize Multi-Tenancy Flink on K8s with YuniKorn
32mA Comparative Analysis of Kueue, Volcano, and YuniKorn
25mSLA Aware Batch Scheduling in Apache YuniKorn with Multi-Tenant Preemption
29mData Processing Efficiency: Optimizing Batch Workloads on Kubernetes With Custom Schedulers
30mBeyond Experimental: Spark on Kubernetes
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
23mPractice of Fine-grained Cgroups Resources Scheduling in Kubernetes