Pilot Execution: Simulating Failure Recovery In Situ for Production Distributed Systems
summary
PILOT implements in‑situ dry‑runs of recovery by spawning phantom threads that shadow‑copy state, propagate a parallel context across RPCs, and cache expensive IO so the real recovery reuses results. The authors reproduce 20 real recovery bugs in HDFS, HBase and others and show PILOT uncovers 17 of 20 failures while baselines catch only eight; runtime overhead is ~3.99% when buggy and ~17.63% end‑to‑end after caching; PILOT also finds and fixes an HBase bug.
AI-written summary · about
chapters (17)
- 0:00Recovery failures in distributed systems
- 1:21Real-world recovery failures
- 2:26Recovery failures in production
- 3:30Offline prevention challenges
- 4:14Parallel execution for simulation
- 5:17Parallel execution challenges
- 6:25In situ simulation with phantom threads
- 7:29Isolating side effects
- 8:12Phantom threads and limits
- 9:16Cross-boundary propagation
- 10:20Condition variables in simulation
- 11:05Caching for parallel execution
- 11:48Pilot system architecture
- 12:31Pilot vs real-world failures
- 13:56Parallel execution performance
- 14:38Bug in latest HBase
- 14:58Conclusion and future
related talks
46mFault tolerance in Distributed Systems: A Case Study in Apache Spark
22mFailing Over without Falling Over
19mProtocol-Aware Recovery for Consensus-Based Storage
57mChaos Engineering Primer
23mExperimental approach towards Replication in Postgres
23mImproving Availability in Distributed Systems with Failure Informers
19mSpeculative Recovery: Cheap, Highly Available Fault Tolerance with Disaggregated...
01h 12mRedundancy does not imply fault tolerance