Why Are Distributed Systems So Hard?
summary
Explains why partitions are inevitable, cites Google data showing 5 rack failures, 3 router failures, 8 network maintenances in a cluster’s first year, and proves CAP forces a choice between availability and consistency during splits. Describes concrete mitigations: leader/follower plus failover, consensus protocols such as Raft, TLS for message integrity, observability, chaos engineering, and design changes for human factors (simpler defaults, fewer alerts) that reduce wrong decisions under operational uncertainty.
AI-written summary · about
chapters (43)
- 0:00Introduction
- 2:07Agenda
- 2:36Storytime
- 3:40Data Evolution
- 4:45Scaling
- 5:22Cloud Computing
- 5:58Why Scale Horizontally
- 7:11What Does It Mean To Run A Distributed System
- 7:49A Node On Distributed Computing
- 8:32Summary
- 9:32Shared Nothing Architecture
- 11:37Unreliable Message Delivery
- 12:36Why Are We Fenced Off
- 12:56Building Observability
- 13:32What We Can Know
- 13:48The Cap Theorem
- 15:22C
- 16:15Replication Lag
- 16:46Consistency is a Spectrum
- 17:31Availability is Not Binary
- 18:46Partition Tolerance
- 21:04Hardware
- 21:24Cables
- 21:35Kevlar
- 21:53Network Partitions
- 22:11Resource Isolation
- 22:35Process Suspension
- 22:46Network Glitch
- 22:56People do bad things
- 23:13Why does this matter
- 23:45The correctness result
- 24:21Mitigation strategies
- 25:15Consensus Algorithms
- 27:26The Woods Theorem
- 27:53Building Mental Models
- 29:22Incident Analysis
- 29:46Blameless Discussions
- 30:15Mental Models
- 31:03Human Failure
- 31:15Alert Fatigue
- 31:40User Mindsets
- 31:55Designing Systems for Humans
- 32:18HugOps
related talks
28mWhy Are Distributed Systems So Hard?
22mWhy Are Distributed Systems so Hard?
27mDistributed Systems, Like It or Not
25mGrowing Distributed Systems
45mDo Distributed Systems they said. It'll be fun they said.
29mHorrors of Distributed Systems
44mDistributed systems and scalability rules
43mWhat We Talk About When We Talk About Distributed Systems