Archive Zone: Lessons Learned
summary
Ceph archive cluster for S3-style objects stores 75M objects using replication 3 across HQ and two sites with ~55 ms latency; initial hardware used flash + big HDDs and two NVMe tiers for DB, but mistakes included splitting the DB disk into 12 parts, creating a pool with only 32 PGs and seeing PG growth of ~34/month, and running Bluestore DB at ~0.83% of OSD capacity (recommend ~4%), which caused metadata issues and OSD restarts; resolved by resizing DB, adjusting PGs, fixing RGW duplication bugs, adding exporters and lifecycle policies (10‑year retention, keep 3 old versions).
AI-written summary · about