The Challenge of Storing Small Objects on a Large Scale
summary
Managing about 100 PiB across 6,000+ OSDs on Ceph exposes two hard facts: millions of tiny email headers blow up storage due to allocation and 4+2 erasure coding (a single kilobyte header can consume ~400 KB), and recovery/backfill for many small objects stalls to days or weeks. Solution stack: multi-blob concatenation with 4 MB target objects, metadata in OMAP anchors on an SSD pool, later rotating 1,000 daily anchors, and moving from 4+2 → 8+3 erasure coding plus Rook/Cephadm and mClock tuning; overall approach works but tombstone and legacy-data backfill issues persist.
AI-written summary · about
chapters (9)
- 0:00Introduction to Proton
- 3:12Early Ceph adoption
- 6:02Small objects problem
- 10:17Multi-blob solution
- 17:36Omaps and challenges
- 19:22Deployment tools evolution
- 26:39Current Ceph setup
- 28:05Growing pains and future plans
- 31:20Q&A
related talks
48mCould Ceph Become More Like a Database?
29mCeph Object Storage Overview, Capabilities and Future Plans
06mI Need More Space, It's Not You It's BlueStore
24mThe Butterfly Effect of a Broken Disk: Top-Down Ceph Troubleshooting To Upstream Contribution
32mCeph Durability: How Safe Is My Data?
43mCeph and Flash
31mAI, ML, and the Ceph Advantage: Scalable Storage for Smarter Workflows
26mSkyhook: Programmable Storage for Databases