YARN Resource Management and Dynamic Max
summary
Uber runs YARN at scale with YRM self-service queue UI that decentralizes queue and capacity edits, enforces permissions, and cuts queue-related tickets by ~90%. Containerization uses a sibling container model (avoiding docker-in-docker), Kraken for image distribution, and reaches ~60% of fleet (~9k nodes). Resource-efficiency features include stress-node detection (marks nodes non-schedulable) and Dynamic Max credit-based algorithm that adjusts queue maxes; reported job throughput improves ~20% and the system prevents queue overheating while preserving min guarantees.
AI-written summary · about
chapters (7)
- 0:00Introduction and team overview
- 6:57YARN resource management improvements
- 13:20Containerization of YARN infrastructure
- 22:23Reliability efforts in YARN cluster
- 25:59Dynamic Max for resource efficiency
- 33:07Scalability challenges and solutions
- 35:58Q&A session
related talks
22mMorcor - Co-Location of Mixed Workloads at Uber
27mHydra: a federated resource manager for data-center scale analytics
43mSystems @Scale Summer Q&A Week 2
38mEnergize Multi-Tenancy Flink on K8s with YuniKorn
40mNomad: Scheduling Applications at Scale
18mReal-time adaptive controls for resilient container workloads
41mIdle Resources Oversubscription: Improving Cluster Utilization
34mPatterns for building Highly Available Cluster Schedulers