Testing Distributed Systems w/ Deterministic Simulation
summary
Claims a deterministic simulation replaces disks, links and machines with transparent software so failures replay exactly; implements a custom concurrency layer as flow C++ extension to run many virtual nodes single-threaded; uses smoke tests like cycle/ring invariants, 5–10 million runs per night (trillions of simulated CPU-hours) plus bugification (randomly alter behavior) to surface timing and coordination bugs. Runs real-world sinkhole hardware too, finds third-party failures (e.g. Zookeeper) and validates that the simulated approach catches real bugs.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 0:24Debugging Distributed Systems
- 1:01A Simple Example
- 2:34Another Simple Example
- 3:15The Real Problem
- 5:54Prerequisites
- 7:14Flow
- 8:25Actor
- 10:18callback junket
- 10:37ring benchmark
- 12:06network simulation
- 13:50Determinism
- 16:03Finding Bugs
- 19:18Other Stuff
- 20:41The Problem
- 21:57Solutions
- 23:05Bugfication
- 24:51Hearst Exponent
- 28:26Simulation Runs
- 28:58Debugging
- 30:38Simulation is Wrong
- 33:00Simulation Cant Test
- 34:31Failures
- 38:14Conclusion
related talks
27mUnderstanding Distributed Systems with Deterministic Simulation
26mCatching Every Bug: Integrating Testing, Observability, Determinism & then what?
38mThe Verification of a Distributed System
40mProperty-testing async code in Rust to build reliable distributed systems
26mSimulant in Anger
15mTesting Consensus Implementations using Communication Closure
25mTesting in production: Ideas, experiences, limits, roadblocks
23mRandomized Testing of Distributed Systems with Probabilistic Guarantees