Idle Resources Oversubscription: Improving Cluster Utilization
summary
Twitter runs a >30,000-node Mesos cluster and finds non‑production jobs hold ~10% of CPUs but exhibit 30–50% idle CPU capacity (p95 of tasks often under 10%). The team builds a Mesos resource estimator and QoS controller that expose idle CPUs as revocable resources, isolate non‑prod in a revocable cgroup, and checkpoint per‑container samples. Implementation works in scale tests with fixes for offer races; production rollout aims to reclaim ~30–40% of non‑prod CPU.
AI-written summary · about
chapters (22)
- 0:00Intro
- 1:01Cluster workload
- 4:19Idle task example
- 4:52Non-production CPU utilization
- 6:31Solution
- 7:21Constraints
- 8:21Oversubscription in Mesos
- 9:36Resource Estimator
- 10:04QoS Controller
- 10:55Idle resources oversubscription
- 12:26Idleness detection
- 13:08Estimated savings
- 14:08Oversubscribable resources
- 16:07QoS correction strategy
- 16:48Recover: idleness preservation
- 18:59Reconstruction - features
- 22:00Workaround
- 22:54Implementation details
- 24:51Configuration
- 26:30Isolation issues
- 28:01Scale issues
- 30:34Current state
related talks
41mEfficient cluster resource management by using Cook and Mesos
41mEfficient cluster resource management by using Cook and Mesos
43mSystems @Scale Summer Q&A Week 2
52mNightmares of a Container Orchestration System
16mTowards Optimal Rack-scale μs-level CPU Scheduling through In-Network Workload...
18mReal-time adaptive controls for resilient container workloads
41mYARN Resource Management and Dynamic Max
38mMesomatic: The Cluster is a Library