Hydra: a federated resource manager for data-center scale analytics
summary
Hydra is a federated resource manager built by extending Apache YARN and deployed across ~250,000 servers to schedules nearly one trillion tasks and process close to a Zettabyte of data. Clusters run up to 50,000 nodes, single jobs reach ~2 million tasks, and Hydra sustains ~70k scheduling decisions/sec by partitioning into 2–5k‑node subclusters and using an AM/RM proxy plus a global policy generator that pushes scheduling policies across tens of thousands of nodes within seconds. Deployment maintains 60–90% CPU utilization, was migrated in place, and ~200k lines of code were open sourced.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:27BIGDATA SCHEDULING: A JOURNEY...
- 3:27HYDRA CHALLENGES
- 4:25THE SCALE/UTILIZATION CHALLENGE...
- 6:02HYDRA ARCHITECTURE
- 9:24SCHEDULING DESIDERATA
- 10:26POLICIES
- 12:54PROPOSED SOLUTION
- 14:52HANDLING GPG DOWNTIME
- 20:48QUALITATIVE EXPERIENCE
- 21:59CONCLUSION
related talks
15mHydra : Resilient and Highly Available Remote Memory
16mHydra: Serialization-Free Network Ordering for Strongly Consistent Distributed Applications
41mYARN Resource Management and Dynamic Max
50mProcessing Billions of Events in Real-Time with Heron
20mAdvancing HPC The Hybrid Scheduler Approach at the D.E.Shaw Group
25m8 Million Tasks Daily: Driving Efficiency in Multi-Tenant Data Platforms at Ho...
18mHorus: Granular In-Network Task Scheduler for Cloud Datacenters
52mProcessing Billions of Events in Real-Time with Heron