Test of Time Award: Analysis of Data Corruption in the Storage Stack

on YouTube ↗09m

" summary

Analysis of telemetry covers 1.5 million drives and records 400,000+ corruptions detected as checksum mismatches. Nearline drives show an order-of-magnitude higher mismatch rate than enterprise disks, with spatial and temporal locality in errors. About 0.66% of some models develop corruption within 17 months of deployment. Results tie specific failure modes to checksums, scrubbing and two-drive protection adoption, and note the same corruption classes remain relevant as SSDs replace HDDs.

AI-written summary · about

" chapters (5)

1. 0:00Start
2. 0:48Pre-2007: Very Little Data on Disk Failures
3. 1:452007-08: Large Scale Studies of Disk Failures
4. 2:18An Analysis of Data Corruption: Highlights
5. 4:41Impact and Relevance

" related talks

• 28mA Study of SSD Reliability in Large Scale Enterprise Storage DeploymentsBianca Schroeder, Kaveh Mahdaviani, Stathis Maneas, Tim Emami · FAST '20
• 16mPerseus: A Fail-Slow Detection Framework for Cloud Storage SystemsErci Xu, Jiesheng Wu, Jiwu Shu, Yiming Zhang, Guangtao Xue, Ruiming Lu, Zhaosheng Zhu, Mengtian Wang, Zongpeng Zhu, Minglu Li, Fengyi Zhu · FAST '23: 21st USENIX Conference on File and Storage Technologies
• 27mMaking Disk Failure Predictions SMARTer!Tirthak Patel, Devesh Tiwari, Weisong Shi, Sidi Lu, Bing Luo, Yongtao Yao · FAST '20
• 49mThe Math of ReliabilityAvishai Ish Shalom · ApacheCon@Home 2021
• 29mCulturing Resiliency with Data: A Taxonomy of OutagesRanjib Dey · Chaos Conf 2020
• 39mBeyond Goldilocks ReliabilityNarayan Desai · SREcon21
• 32mDebugging Data Corruption in PostgreSQL: A Systematic ApproachPalak Chaturvedi, Nitin (mentioned) · POSETTE: An Event for Postgres 2025
• 01h 01mThe Performance Investigator's Field GuideSasha Goldshtein · NDC London 2018