Fast Erasure Coding for Data Storage: A Comprehensive Study of the Acceleration...
summary
Shows a concrete pipeline: search bitmatrix by simulated annealing/genetic algorithms to minimize a weighted cost, then generate a schedule that applies XOR reduction, cache-aware updates and XOR-level vectorization. Reports numbers: 130% throughput improvement from XOR-vectorization, ~35% fewer XORs from normalization/matching, ~25% fewer XORs from smart scheduling, ~5% throughput gain from cache tweaks. Implements on Jerasure with SSE128, matches or beats EVENODD/RDP/STAR and often doubles throughput vs direct finite-field vectorization; code is released.
AI-written summary · about
chapters (24)
- 0:00Intro
- 0:55Data Reliability
- 1:13Replication - Erasure Code
- 1:48Acceleration Techniques
- 5:41Cauchy Reed-Solomon Codes
- 6:15Fast GF Ops - Binary Representation
- 7:27Techniques Tiers
- 7:46Bitmatrix Normalization (BN)
- 8:21Smart Scheduling (SS)
- 9:49Scheduling - Cache Optimization (S-CO)
- 10:43Vectorization
- 11:55Question to Answer
- 12:37Optimization Tiers
- 13:15Combinations (ij)-strategy
- 13:52Choice of Cauchy Matrix
- 17:24Cost Function Improvement
- 18:10Proposed Coding Procedure
- 18:40Testing Setup
- 19:26Encoding v.s. Efficient RS/CRS code
- 20:03Encoding v.s. Three Parities Codes
- 20:26Encoding v.s. Two Parities Codes
- 20:41Overall Encoding Improvement
- 21:30Decoding Throughput
- 21:50Conclusion
related talks
01h 14mErasure Coding at Scale
01h 07mEmerging Embdedded Memories and Storage
32mUnder the Hood: Key Algorithms Powering Vector Search
22mELECT: Enabling Erasure Coding Tiering for LSM-tree-based Storage
12mWhy is your encoder so slow? The curse of next-gen standards
27mFPGA-Accelerated Compactions for LSM based Key-Value Store
39mWhat's in a Vector Database?
27mPreview: Storage