Erasure Code at Scale

on YouTube ↗32m

" summary

Production Ceph cluster named ECHO provides erasure coded object storage for LHC workloads using large storage nodes (30+ drives) and 8 data + 3 parity stripes for k+m protection, initially ~13 PB raw and ~10 PB live, achieving sustained throughput and spikes past over 20 GB/s production IO. Implementation uses librados striper with 64 MB stripes (8 MB on-disk chunks). Early Jewel-era peering and backfill bugs, pending-sector driven inconsistent PGs, and manual crushmap edits create operational work but upgrades (async messenger, Luminous) and tuning make the design viable.

AI-written summary · about

" chapters (18)

1. 0:00Intro
2. 1:03WLCG
3. 2:14Grid Storage
4. 3:48Ceph
5. 5:44Data Flow
6. 7:03Storage
7. 10:00Data Pool
8. 13:02Availability
9. 14:04Object Sizes
10. 16:22Does it work
11. 18:18Backfilling
12. 19:35Operational Issues
13. 22:09Inconsistent Placement Groups
14. 24:22Pending Sectors
15. 25:55Manual Cluster Management
16. 27:09Rolling Back Changes
17. 28:38Data Distribution
18. 30:03Conclusion