Ceph Operations at Scale
summary
DigitalOcean runs Ceph as primary block and object storage across 75 clusters totaling ~270 PB and ~34,000 OSDs on 2,000 nodes, so automation focuses on avoiding SSH and human error. Key pieces are a se-tools shim that makes containerized Ceph appear like packages, an Ansible-based storage CM for opinionated deploys and CRUSH generation, a disk lifecycle service with central DB + node daemon and rados locks for safe gated transitions, automated fleet reboots to catch bad disks, and observability via Maragraph, store exporter and object logger.
AI-written summary · about
chapters (12)
- 0:00Introduction to talk
- 0:22About DigitalOcean and Ceph
- 1:50Challenges with legacy Ceph setup
- 3:14Containerizing Ceph for scalability
- 5:44Automation tools and Storage CM
- 7:55Cluster lifecycle and maintenance
- 9:40Automated node reboots and safety
- 11:46Disk lifecycle automation
- 14:14Safety gates in automation
- 15:41Benefits of automation at scale
- 16:21Ceph observability tools
- 21:28Q&A session
related talks
49mIntro to Ceph, the Distributed Storage System
02h 59mCeph Storage: a Caffeinated Primer
38mTesting Ceph: Pains & Pleasures
35mMaking Ceph Fast in the Face of Failure
30mScaling Databases and File APIs with Programmable Ceph Object Storage
29mGlobal Deduplication for Ceph
12mCeph QoS: How to support QoS in distributed storage system
32mStretch Clusters in Ceph: Algorithms, Use Cases, and Improvements