Horrors of Distributed Systems
summary
Covers concrete failure modes like bit flips in SSDs and RAM (Google finds ~1 bit flip/GB every two hours on non‑ECC RAM) and networking limits (Melbourne→us‑east1 ≈16,000 km → ~50 ms one‑way, ~100 ms roundtrip). Describes engineered choices: channels uses at‑most‑once semantics while Kafka targets at‑least‑once and explains costs of each, shows real bugs (Celery tasks racing replica lag, webhooks firing before replication) and simple fixes such as delay webhooks by five seconds to avoid inconsistency.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:45About me
- 1:38Theyre really hard
- 2:14Nonbinary failure
- 2:26The spectrum
- 2:55Discs and RAM
- 5:33Network
- 6:46Bad Neighbors
- 8:07Bugs in Networking
- 8:26Speed of Light
- 9:35Australia
- 11:01Wait Period
- 11:57Time
- 12:46Time Delay
- 14:41Communication
- 16:29Kafka
- 17:35Idempotent
- 18:53Events
- 19:06Consensus
- 20:18Replication Lag
- 23:05The SEEP Theorem
- 24:21Tradeoffs
- 24:40Interfaces
- 25:33Design
- 27:11Conclusion
related talks
34mDistributed Systems - What Can Go Wrong and How to Get It Right
27mDistributed Systems, Like It or Not
25mGrowing Distributed Systems
33mWhy Are Distributed Systems So Hard?
52mDistributed Computing Dealing with Time and Failure in the Wild
28mWhy Are Distributed Systems So Hard?
31mWhat Breaks Our Systems: A Taxonomy of Black Swans
02h 42m!!! Do not publish or share !!!!