Get That Cluster Back Online, but Hurry Slowly
summary
Pawsey’s petabyte Ceph RGW cluster suffers a months-long degradation with one cluster unhealth for seven weeks, OSDs holding 50–90 PGs per OSD instead of the recommended 100, and RGW garbage collection growing to millions of GC entries; repeated RGW freezes, OOMs and slow uploads follow. Recovery actions include controlled use of the upmap remap script, reverting legacy reweights, targeted backfilling and one crucial OSD restart that restores service; outcome: data intact, balance and scrubbing resume, and monitoring, runbooks and post‑mortems are improved.
AI-written summary · about