Fault tolerance in Distributed Systems: A Case Study in Apache Spark
summary
Apache Spark uses lineage-based recovery instead of replication, which keeps the fast common case but forces recomputation across multiple shuffle stages when a task or partition is lost. Real failures include one bad disk taking out a 1,000-node cluster because tasks repeatedly reschedule on the flaky node, an infinite retry bug that masked failures, and fragile fault-injection testing coverage. Fixes include thread-local workarounds, new heuristics around retries and speculative execution, and heavier testing for rare timing windows.
AI-written summary · about
chapters (8)
- 0:00Fault tolerance intro
- 8:24Spark’s lineage model
- 13:20Failure types
- 20:00Hardware corruption
- 24:37Spark error handling bugs
- 30:15Fault injection testing
- 33:47Fault tolerance lessons
- 37:03Q&A
related talks
01h 12mRedundancy does not imply fault tolerance
36mDesigning for failure - architecting fault-tolerant system
52mDistributed Computing Dealing with Time and Failure in the Wild
31mRethinking of fault tolerance in Flink: what lies ahead?
15mA Multiparty Session Typing Discipline for Fault-tolerant Event-driven Distributed Programming
40mProductionizing Big Data - Stories from the Trenches
38mFault Tree Analysis Applied to Apache Kafka
02h 18mDistributed Systems: In the Daily Life of the Data Engineer