Tackling operational and capacity utilization challenges at a large scale enterprise deployment
summary
Workday runs >400K cores and ~7K hypervisors across 25 clusters (growing toward 10K) and addresses capacity by simulating placements with a JavaScript tool, validating Git + Gerrit + Jenkins-driven deployment plan changes, and changing the Nova scheduler with waivers and a memory multiplier plus soft-affinity to balance fragmentation and blast rate. Results show large reductions in fragmentation, a lower maximum blast rate, fewer hosts needed to satisfy big VMs, and the ability to empty hosts for weekly patching via uptime weighting.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:23With bigger clouds comes tougher challenges
- 5:36Fragmentation
- 7:28Host Aggregates
- 9:40Simulation tool
- 11:03Workday Scheduler Setup
- 12:46Results - cluster 1
- 13:18Maintenance aware weigher
- 15:20How to understand the weight combination* Hosts
- 16:23Challenges and requirements
- 18:29Self-service...nice to have
- 19:40Gerrit + Jenkins capacity management workflow
- 20:41Check Current Capacity
- 22:32Jenkins Artifacts Archive
- 22:54Cluster Visualization
- 23:26Lessons learned
related talks
47mScaling FinOps with a holistic approach to resource optimization
33mHumana - Scaling Cloud Foundry From One Team to The Entire Enterprise
30mHow Workload Rightsizing Increases Availability
43mSupporting NASA Science with OpenStack
27mFrom Supercomputing to Serving: A Case Study Delivering Cloud Native Foundation Models
48mCloud FinOps & Kubernetes Optimisation at Scale
21mExperience Operating Large GPU Clusters at Organizational Scale
14mHyperscale Infrastructure Q&A