Fault Tolerance on the Cheap Making Systems That Probably Won't Fall Over
summary
Claims a real system handles 40 million transactions a second in a tenth of a second using airling and practical patterns rather than perfect proofs. It shows concrete techniques: supervise and restart graphs of components, immutable data and isolated side effects, compile-time guarantees and proofs for hot spots, redundant clusters (no single point of failure), instrumentation that treats S3 as a yearly failure mode, and organizational processes that turn on-call fixes into long‑term repairs.
AI-written summary · about
chapters (30)
- 0:00Intro
- 0:41I do things to/with computers.
- 2:42Go program a computer!
- 3:21FAULT TOLERANCE
- 3:39Sub-components fail. The system does not
- 8:24Intentionally stifles creativity.
- 11:14Move fast and break things!
- 18:56The most atomic level of the system.
- 19:32Immutable Data Structures
- 20:20Isolate Side-Effects
- 21:22Compile-Time Guarantees
- 22:21Why test, when you can prove?
- 23:07This is Functional Programming
- 24:17components are
- 24:30interactions are
- 25:09Supervise, and restart.
- 26:02Use only addressable names.
- 26:52Distinguish your critical components.
- 28:01CLUSTER
- 28:27Redundant Components
- 29:34No Single Points of Failure
- 30:40Mean Time to Failure Estimates
- 31:20Instrument and Monitor
- 32:17A finely built machine without a supporting organization is a disaster waiting to happen.
- 32:30Deepwater Horizon Asiana #214 Magnitogorsk Therac-25 Damascus Incident New Orleans Levee
- 36:15Process is Priceless
- 37:07Build flexible tools for experts.
- 38:46Build with Failure in mind.
- 42:14Every system carries the potential for its own destruction
- 44:570. The network is unreliable. 1. Latency is non-zero. 2. Bandwidth is finite. 3. The network is insecure. 4. Topology changes. 5. There are many administrators. 6. Transport cost is non-zero. 7. The network is heterogenous.
related talks
52mPerformance & Stability Anti patterns
16mEngineering Failure-Resilient Systems
43mGetting Your System into Production and keeping It There
01h 02mFive things every developer should know about building mission-critical systems
31mRadical Ideas Enterprises Can Learn From The Cloud
10mDesigning for Failure using AWS: Resilient Systems
22mResilience Is Not Enough
10mBuilding an 'Oops-Less Operation' at Scale