Multi Cluster Ceph at Scale
summary
Runs and manages 23 clusters across five data centers with more than 1,200 servers and almost 100 PB raw using Ceph RBD/S3, containerized services and safadm; moves MySQL backups from NFS to Ceph cutting backup time from ~24h to ~7h and serving ~2 PB during backups. Uses erasure coding (K2/M4) for high-density racks, faces rebuilds that take days–weeks, sees 3–16 drive failures/month across ~20,000 drives, and relies on CI/CD, automated onboarding and more automation to scale while upgrades and cross-site replication remain work in progress.
AI-written summary · about
chapters (9)
- 0:00Introduction and context
- 3:23Why multiple Ceph clusters
- 6:16Cluster types and hardware
- 10:00Growth and scale challenges
- 13:12Continuous delivery setup
- 16:10Customer onboarding automation
- 20:33Lessons learned and community
- 23:24Upgrade and maintenance hurdles
- 26:37Future improvements and automation
related talks
37mOne Cluster to Rule Them All - Multi-Tenancy for 1k Users
06mSponsored Keynote: What I Learned Running 10,000+ Kubernetes Clusters
36mResilient Multi-Cloud Strategies: Harnessing Kubernetes, Cluster API, and Cell-Based Architecture
13mSponsored Keynote: Multi What???? Multi-cloud, Multi-cluster, Multi-tenanted K8s
35mAuto Operation of Hundreds of K8s Clusters in Yahoo! Japan
39mManaged CF-as-a-Service: An Operator's Cookbook
20mReliable Migration Of 4+ Million CPU Cores of Compute to Kubernetes
14mKeynote: The Big Leap: Conquering the Legacy System and Building a Scalable Cloud Infra...