Erasure Coding at Scale
summary
YT runs over 1 EB of data and applies erasure coding to both immutable blobs and append‑only journal chunks, combining Reed‑Solomon (3+3) with write quorum 5 read quorum 4 to tolerate two node losses while halving storage/network overhead compared with 5x replication (encoding overhead ≈2x for that config). Implementation uses ISA‑L library (jerasure caused misaligned parity), uses recovery striping for journals, speeds node decommissioning, reduces cross‑DC traffic, and requires throttlers and concurrency limits to avoid repair storms.
AI-written summary · about
chapters (12)
- 0:00Start
- 5:10e Overview
- 7:16ata and Control Plane
- 15:05acement Pros and Cons
- 17:58Types of Failures
- 23:20hunk Structure
- 39:24I Chunks: Contract
- 43:29I Chunks: Read Pipeline
- 49:43g vs Striping
- 53:04Practical Scenario
- 54:34argument
- 56:04rils of Jerasure
related talks
11mStripeless Data Placement for Erasure-Coded In-Memory Storage
15mLESS is More for I/O-Efficient Repairs in Erasure-Coded Storage
24mOpenEC: Toward Unified and Configurable Erasure Coding Management...
22mELECT: Enabling Erasure Coding Tiering for LSM-tree-based Storage
11mExploiting Combined Locality for Wide-Stripe Erasure Coding in Distributed Storage
59mPractical Error Control Coding For Data Storage
38mUsing Erasure Codes to Enable Interactions over Networks
26mFast Erasure Coding for Data Storage: A Comprehensive Study of the Acceleration...