Unlocking unallocated cloud capacity for long, uninterruptible workloads
summary
Characterization of Harvest VMs reveals high variance in interchange times; the system uses distribution-based predictions of stability plus a scheduling and a resource-acquisition component to match long, hard-to-checkpoint tasks to stable HVMs and maintain a stable pool by retiring recently changed VMs. Implemented inside YARN and evaluated with production HVM traces and genomics and seismic workloads, the approach cuts mean job completion time by 27%, 90th-percentile by 44%, and runs at roughly 75% lower cost than regular VMs.
AI-written summary · about
chapters (11)
- 0:00Unallocated cloud capacity intro
- 1:28Harvest VMs background
- 3:17Challenges on Harvest VMs
- 5:25Introducing SlackShed
- 8:35Scheduling on stable VMs
- 10:19Predicting resource stability
- 11:45Maintaining stable VM pool
- 12:29System design and implementation
- 13:12Scheduling improvements evaluation
- 14:14Resource acquisition benefits
- 14:59Summary
related talks
32mResearch Workloads on Preemptible VMs
25mTackling operational and capacity utilization challenges at a large scale enterprise deployment
15mExperiences in Managing the Performance and Reliability of a Large-Scale Genomics...
32mHVX: Virtualizing the Cloud
19mStarburst: A Cost-aware Scheduler for Hybrid Cloud
30mThinner Clouds with Preallocation
15mHyrax: Fail-in-Place Server Operation in Cloud Platforms
30mCapacity Management for Fun & Profit