Keeping Ceph RGW Object Storage Consistent
summary
Bloomberg runs Ceph RGW across tens of clusters and four data centers serving >2 billion S3 requests/day and 77 PB raw; they see orphan multipart parts, version-epoch races that caused data loss, datalog entries lost when rgw caches entries in memory, and unfinished bucket-index transactions. Proposal implements bucket-level scrubbing as background jobs that record progress, run configurable extensible modules to detect missing tail/hat objects, lost log entries and replication mismatch, and offer report and auto-repair modes. Prototype uses existing rgw admin commands, cross-zone S3 fetch + ETag verify, and starts with multisite replication checks and benchmarking.
AI-written summary · about
chapters (16)
- 0:00Introduction to talk
- 0:22Talk agenda overview
- 1:05Who we are Bloomberg
- 2:35Storage engineering team focus
- 3:38RGW cluster scope details
- 4:01RGW data layout explained
- 6:10Causes of data inconsistency
- 7:53Inconsistency scope and consequences
- 8:14Case one multipart uploads
- 11:30Case two version updates racing
- 14:05Case three crash consistency
- 19:07Other inconsistency cases
- 20:51Need for active consistency checks
- 21:33Bucket scrubbing solution
- 27:14Future plans and community effort
- 28:19Q&A session
related talks
26mImplementing a Resilient Ceph RGW Object Storage with Erasure Coded Data Pool as the Foundation...
18mClient Driven Geo-Redundancy for Ceph Object Storage
02h 59mCeph Storage: a Caffeinated Primer
27mFrom Chaos To Clarity: Systematic Ceph Cluster Analysis for Rapid Troublesho...
34mModernizing Ceph Deployments at CERN: CephFS and Object Storage Across Data Centres
29mGlobal Deduplication for Ceph
49mIntro to Ceph, the Distributed Storage System
35mMaking Ceph Fast in the Face of Failure