200 Clusters vs 1 Admin
summary
OVH runs ~210 Ceph clusters totalling ~50 PB on ~2,000 servers and 25,000 HDDs, and keeps one admin effective over 200 clusters by building a centralized Python control plane backed by MySQL Galera and a Puppet provisioner. Clusters run OSDs in containers (Docker/LXD), use flashcache + NVMe, and orchestration uses Celery workflows for sequential and parallel operations. Automated PG growth, rack-by-rack upgrades, weighted OSD drain and automatic disk/NVMe replacement provide self‑healing and scalable management.
AI-written summary · about
chapters (22)
- 0:00Intro
- 1:14A few words about OVH
- 2:06Using Ceph @ OVH
- 3:04Ceph in numbers @ OVH
- 6:51Typical Ceph setup @ OVH - block devices
- 9:03Ceph inside containers?
- 10:06Typical Ceph setup @ OVH - whole story
- 11:30Typical Ceph setup @ OVH - design goals
- 12:11Multiple Ceph clusters - Advantages
- 14:23Multiple Ceph clusters - Drawbacks
- 15:57Ceph as a Service - Control Plane
- 16:19Control Plane - What should it manage?
- 17:37Control Plane - What does it do?
- 18:16Control Plane - How is it built?
- 19:11Control Plane - Ceph management
- 20:20Control Plane - Cluster management
- 21:35Control Plane - External API
- 22:54Control Plane - Maintenance Operations
- 28:04What's next?
- 28:37Monitoring Ceph
- 29:19The Holy Grail - Self Healing
- 29:46Self Healing - Example Remove disk with I/O errors
related talks
28mA Welcoming Ceph Experience Improvements for First-Time Users or: How to Make Beginners Love Ceph
29mTowards Sustainable Multi-Cluster Management in Real-World Academic Infrastructure
32mMonitoring OVH: 300K servers, 27 DCs one unified metrics platform
41mMonitoring OVH: 300k Servers, 27 DCs, a Metrics Platform
16mTips and Tricks for Managing and Administering Ceph Clusters
52mLet’s build a private cloud – how hard can it be?
03mFrom 5 to 1,300+ Clusters: Declarative Scaling for Private Kubernetes
38mDeploying Database Clusters in the Cloud