Distributed Databases: What Could Possibly Go Wrong?
summary
Distributed databases fail because of concrete tradeoffs and physics not platitudes: a DNS glitch inside DynamoDB on Oct 20, 2025 makes Dynamo unreachable for 3 hours, triggers EC2 control plane errors and produces 17 million outage reports with up to 15 hours of disruption; Cloudflare’s Nov 18, 2025 permissions change causes 3.5 hours of global outage. Coverage includes Raft/Paxos quorums, ~200 ms intercontinental RTTs, Yugabyte/Cosmos/Spanner examples, and practical fixes like transactional outbox, idempotency keys, and shard key selection that were built as resilient patterns.
AI-written summary · about
chapters (13)
- 0:00Distributed databases intro
- 1:57Simple mental model
- 5:01Three tradeoffs
- 8:10Eight fallacies
- 10:58Understanding replication in distributed databases
- 15:12Consistency models
- 20:03Consistency failure example
- 21:05Network partitions
- 22:52Sharding strategies
- 26:42Blast radius failures
- 28:08Resilient patterns
- 36:53Design questions
- 40:04Summary
related talks
57mDSS Keynote — Back to the Future: Prepare Your Data Infrastructure for Anything
20mMaximise database availability
26mThe tools and techniques behind resilient systems
01h 00mDistributed transactions implementation trade-offs
43mDYNAMO: the paper that changed the database world
28mWhy Are Distributed Systems So Hard?
40mThe World Blew Up but We’re All Okay: How We Managed a...
24mFailure is Not an Option: Highly Available Distributed SQL