Averting crisis at multi-petabyte scale
summary
OVH scales down massive pools to limit blast radius: split 1PB pool into four 250TB clusters so failures affect fewer customers and rebuild work is smaller — recovery reduced from 514TB to 135TB in the example shown. Investigation finds a cron-like image clone/remove loop every five minutes, so ops increase logging, trace client IPs, and pin the OpenStack image cloner as the culprit — cron job causing image clone/remove every 5 minutes. A custom repair manipulates binary RADOS keys via librados C API to fix RBD children entries (patch targeted at Luminous).
AI-written summary · about
chapters (9)
- 0:00Introduction to crisis at scale
- 1:03Scaling issues with petabyte clusters
- 3:56Human errors and phantom workloads
- 5:02Debugging unexpected cron job activity
- 8:02Investigating phantom image clones
- 9:06Tackling unexpected disk space usage
- 12:25Broken parent-child image links
- 17:25Manual fixes and community help
- 18:26Q&A on scaling strategies
related talks
29mBuilding Resilient Systems
38mEfficient Ceph Performance Troubleshooting in Production Using eBPF
25mWe're Still Down: A Metastable Failure Tale
15mWhy We Have 17 Clusters Instead of One (and Why You Probably Shouldn’t Do Th...
27mFrom Chaos To Clarity: Systematic Ceph Cluster Analysis for Rapid Troublesho...
35mMaking Ceph Fast in the Face of Failure
34mDistributed Systems - What Can Go Wrong and How to Get It Right
33mCase Study : Improving Resilience of Applications in Telco Environme...