Redundancy does not imply fault tolerance

on YouTube ↗01h 12m

" summary

Measured eight popular distributed storage systems by injecting a single-block fault at a time (read corruptions, read/write errors) using a user-level FUSE injector ErrorFS, and finds that one faulty block on one node often causes silent user-visible corruption, data loss, unavailability, or spread of corruption rather than recovery from replicas. Root causes are undetected local faults, crash-or-truncate reactions, and unsafe local/global interactions. Built a protocol-aware recovery called control, applied to LogCabin and ZooKeeper, which remains safe and highly available with only 4–10% throughput overhead.

AI-written summary · about

" chapters (37)

1. 0:00Intro
2. 0:26Distributed Storage Systems
3. 0:36Redundancy for Fault Tolerance
4. 1:18Redundancy in Distributed Storage
5. 1:54Storage Faults Are Prevalent
6. 4:10Redundancy Does Not Imply Fault Toler
7. 6:17Fundamental Problems: Summary
8. 7:49Outline
9. 9:29Fault Model: ext4 and btrfs
10. 10:16Fault Injection Methodology
11. 11:47Behavior Inference Methodology
12. 12:28An Example: Redis
13. 13:42Redis: Behavior Analysis
14. 18:38Faults are often Undetected Locall
15. 20:30Crashing - Common Local Reaction
16. 21:54Crash and Corruption Handling are Enta
17. 28:48How to Recover Faulty Data?
18. 36:25RSM Overview
19. 37:26Replicated State Update
20. 38:46RSM Persistent Structures
21. 40:01Current Approaches to Handling Fault
22. 40:41Protocol-Oblivious: Crash
23. 44:50Recovery Approaches Summary
24. 46:14CTRL Overview
25. 46:47CTRL Fault Model
26. 47:26CTRL Guarantees
27. 47:53CTRL Local Storage
28. 48:51CTRL Distributed Recovery
29. 49:12Properties of Consensus Protocols
30. 50:37Follower Log Recovery
31. 52:22Leader Log Recovery: Determining Commitment
32. 53:14Leader Recovery: Determining Commi
33. 56:15Evaluation
34. 56:47Reliability Experiments Example
35. 57:34Reliability Experiments Summary
36. 58:28Reliability Results Summary
37. 58:50Update Performance (SSD)

" related talks

• 23mCan Applications Recover from fsync Failures?Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Anthony Rebello, Yuvraj Patel · USENIX ATC '20
• 19mEvaluating File System Reliability on Solid State DrivesBianca Schroeder, Shehbaz Jaffer, Stathis Maneas, Andy Hwang · USENIX ATC '19
• 46mFault tolerance in Distributed Systems: A Case Study in Apache SparkImran Rashid · Scala Days Copenhagen 2017
• 36mDesigning for failure - architecting fault-tolerant systemJ. Derda · 4Developers 2015
• 18mWho's Afraid of Uncorrectable Bit Errors? Online Recovery of Flash Errors with Distributed RedundancyMichael J. Freedman, Kyle Jamieson, Amy Tai, Asaf Cidon, Andrew Kryczka, Shobhit O. Kanaujia · USENIX ATC '19
• 19mProtocol-Aware Recovery for Consensus-Based StorageEric Lee, Aishwarya Ganesan, Aws Albarghouthi, Ramnatthan Alagappan, Vijay Chidambaram, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau · USENIX ATC '19
• 15mFiDe: Reliable and Fast Crash Failure Detection to Boost Datacenter CoordinationPavel Chuprikov, Patrick Eugster, Patrick Jahnke, Davide Rovelli, Philipp Berdesinski, Ali Pahlevan · USENIX ATC '25
• 17mOne-Size-Fits-None: Understanding and Enhancing Slow-Fault Tolerance in Modern...Yuxuan Jiang, Peng Huang, Guangtao Xue, Ruiming Lu, Yunchi Lu · NSDI '25