The Art of Embracing Failure at Scale
summary
Explains concrete blast‑radius reductions: use bulkheads and isolation to limit faults, run services in many cells (thousands to tens of thousands) routed by Route 53, and apply shuffle sharding and small shard sizes to drop affected customers from 100% to ~3.6% in the eight‑node/size‑2 example. Mentions DynamoDB and Aurora hitting massive scale (Aurora ~40 million RPS), enforces token‑bucket throttling, exponential backoff with jitter, and chaos tools for deliberate failures; these patterns are deployed at AWS and shown to work in production.
AI-written summary · about
chapters (8)
- 0:00Intro
- 7:10AWS in 2008
- 9:28Regions
- 10:30Availability Zones
- 23:12Shuffle Sharding
- 27:02Back Off
- 29:40Throttling
- 33:50The lethals law
related talks
10mDesigning for Failure using AWS: Resilient Systems
32mResiliency and Availability Design Patterns for the Cloud
42mPlanning To Fail
40mSh... Fail Happens: Fail-aware Events Processing on AWS
42mChaos Engineering: Why Breaking Things Should be Practiced
32mScaling Done Right
51mPatterns for building resilient software systems
16mBlast Radius Reduction for Large-Scale Distributed Systems