Efficient cluster resource management by using Cook and Mesos
summary
Cluster runs on Mesos + Cook to manage tens of thousands of simulation tasks daily; Cook implements preemption using a score based on cumulative dominant-resource share (CRS) so the system reclaims resources in minutes instead of hours. Benchmarks on a seven‑day trace show 30% interactive speed‑up while about 10% large batch slow down, and effective utilization drops only 2–3%. Cook and Mesos are production‑stable (Mesos >3 years, Cook >1 year at Two Sigma) and the matcher/rebalancer pair enforces fair, importance‑aware preemption.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 2:20What does Mesos do
- 4:27Mesos example
- 5:59Lifecycle management
- 7:49Frameworks
- 8:25Two Sigma
- 9:27Simulation workload
- 9:46Simulation use cases
- 11:43Fairness
- 14:45Recap
- 17:17preemption
- 17:48intuition
- 19:45score function
- 22:41CRS
- 25:57Architecture
- 28:01Benchmark
- 32:58Questions
related talks
34mSimulating Our Apache Mesos Framework for Fun and Profit and...
41mIdle Resources Oversubscription: Improving Cluster Utilization
29mGetting Started with Apache Mesos
38mMesomatic: The Cluster is a Library
27mKeynote Panel: Future of Cluster Management
23mHow cluster managers affect the landscape of modern distributed computing
59mBuilding Distributed Systems with Apache Mesos
39mHow we learned to love the Data Center Operating System