Scaling Bare Metal Provisioning with Nova and Ironic at CERN
summary
CERN runs Nova plus Ironic to provision physical servers and now manages 5,000 nodes -> 10,000; scaling hits are concrete not conceptual. iSCSI deploys cause conductor OOMs so CERN adds wing controllers and moves toward direct or Ansible deploys, and wraps ipmitool with an ipmi proxies proxy for flaky BMCs. API slowness comes from disabled pagination and a 60s inspector sync so re-enabling pagination and lengthening sync plus introducing an inspector leader election fixes latency. Resource tracker cycles drop from ~3 hours to minutes by partitioning with conductor groups (~500 resources each).
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:19CERN and CERN IT in 1 Minute ... → Understand the mysteries of the universe!
- 1:23The 1% CERN IT Data Centers
- 1:42CERN's Nova Deployment
- 3:36Nova Instances as Nova Compute Nodes
- 4:11CERN's Ironic Deployment: Node Growth
- 4:47Day-to-day issues we struggle with ...
- 6:52Controller crashes
- 8:15API Responsiveness (3)
- 11:00Resource Discovery
- 11:45Initial Nova / Ironic Setup at CERN
- 12:57The Resource Tracker is slooooow
- 14:38Conductor Groups configuration
- 15:19Nova / Ironic Setup at CERN with Conductor Groups
- 15:36The Transition Steps
- 16:31Impact on the Resource Tracker Cycle Time
- 16:48Number of Resources per Conductor Group
- 17:13Conclusion and Outlook
related talks
15mThe CERN Cloud Infrastructure: 5 years in 15 minutes
20mCERN OpenStack User Story
01h 47mDistributed Power (Full Day 2 includes CERN, Expedia, TapJoy and Telco Panel)
44mLINE's Journey; Road to 4 Million Cores in t...
32mAutoscale Your on-Premises Bare Metal Kubernetes Clusters
35mOptimising resource allocation in hyperconverged infrastructure
41mMetal As A Service — Manage Your Bare Metal in MaaS Like a Virtual Machine
01h 50mKeynote