A 10-Year Retrospective Operating Ceph for Particle Physics
summary
Started with a 300 TB proof of concept in 2013 and grows to ~100 PB raw across 17 Ceph clusters, with an early jump to 3 PB in production; experiments include erasure coding, multi-datacenter S3, and large-scale 10k OSD stress tests. BlueStore adds end-to-end checksums; switching drives to write-through mode cuts latency from ~75 ms to 3–5 ms on bulk clusters. A faulty lz4 compression change causes an eight-hour outage, so clusters are split into availability zones; EOS and tape handle the physics data while Ceph hosts core infrastructure.
AI-written summary · about