Cluster storage systems gotta have HeART...
summary
An online tool called HeART monitors per–make/model annualized failure rates, uses an anomaly detector (robust random cut forest) and a change-point detector (ruptures) to find bathtub-curve knees, then picks cheaper erasure-code parameters that still meet a target MTTDL. Applied to Backblaze’s 100k+ drive traces, it re-encodes groups (e.g., h4a from 6-of-9 to 12-of-15) and reports 11–33% space savings overall while remaining safe, online, and constrained to modest stripe widths to limit reconstruction cost.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:50Reliability heterogeneity in disks
- 1:08Overview of exploiting reliability heterogeneity
- 2:11Cluster storage system reliability • Failures common in today's cluster storage systems
- 5:27The bathtub curve (each disk group)
- 6:27Two disk groups over time
- 7:20Disk-group reliability timeline
- 7:52Heteretogeneity-Aware Redundancy Tuner
- 8:29AFR in useful life: stability & anomalies
- 10:31Change point detection
- 11:14Leveraging existing algorithms
- 12:06The Backblaze dataset
- 12:59Methodology to evaluate cost reductions
- 15:03Evaluation on Backblaze dataset . 5-4 disks have the highest AFR (4.01%) in Backblaze
- 15:54Disk group AFR
- 16:23HeART in action: H-4A HDDs
- 19:00Storage savings: 6-of-9
- 21:16Conclusion • Exploiting reliability heterogeneity reduces storage cost
related talks
39mDon’t eat my data - 30+ years of storage war stories
53mOvercome Blast Radius Anxiety With Ultrafast Rebuilds and Performance for High-capacity SSDs
14mRe-think Data Management Software Design Upon the Arrival of Storage Hardware with
01h 12mRedundancy does not imply fault tolerance
45mBig Data and SSDs
29mData @Scale Seattle - Kestutis Patiejunas
28mData @Scale Seattle- Brian Beach
41mDesign (Failures) Case Study