Fractal: Fault-Tolerant Shell-Script Distribution
summary
FRACTAL implements fault-tolerant distribution for unmodified shell scripts by treating each worker-assigned fragment as an opaque subgraph and tracking only its input/output boundaries with subgraph-level progress tracking and remote pipes that report byte offsets to a centralized monitor. It re-executes only affected subgraphs, persists outputs selectively, and enforces exactly-once semantics. On 4- and 30-node clusters across 77 scripts (up to 33.4 GB), FRACTAL achieves >9.6x speedup over Bash, >5.5x over Hadoop Streaming, 17% over DiSh, and recovers 7.8–16.4x faster than Hadoop Streaming.
AI-written summary · about
chapters (5)
- 0:00Fractal introduction
- 2:44Fault-tolerant shell pipelines
- 4:49Fractal fault tolerance
- 8:41Performance benchmarks
- 12:35Summary and open-source
related talks
10mDiSh: Dynamic Shell-Script Distribution
38mFault Tree Analysis Applied to Apache Kafka
01h 12mRedundancy does not imply fault tolerance
46mFault tolerance in Distributed Systems: A Case Study in Apache Spark
25mStretch: A Fault-Driven DSM Runtime for Distributed Multithreaded Applications
19mCAFault: Enhance Fault Injection Technique in Practical Distributed Systems via...
50mBecoming antifragile is more important than ever in disruptive times
53mPanel Discussion: Fragmentation of the Batch Ecosystem in Kubernetes, Challenges and Solutions