Performance & Stability Anti patterns
summary
Shows concrete anti-patterns and fixes: use Polly for retries, exponential backoff and circuit breaker policies; always set timeouts and avoid trusting internal inputs. Cites Google SRE finding that ~70% of outages follow live changes and a study showing 90% of catastrophic failures stem from bad error handling (77% reproducible by unit tests). Describes bulkheads, queue-based load shedding, game days/fault injection, Datadog/CloudWatch style metrics, and Xero’s cell model (multiple isolated cells) as practical, working mitigations.
AI-written summary · about
chapters (27)
- 0:00Start
- 5:15Possible Causes of Failure
- 8:42Trade-offs
- 9:20Foundations
- 10:29Reactive Manifesto
- 12:29Availability Cost/Benefit
- 14:04Mean Time to Failure (MTTF)
- 14:26Mean Time to Recovery (MTTR)
- 14:45Other Considerations
- 16:40Flood of Fail
- 19:05Bulk Heads
- 19:42Deutsch's 8 fallacies of distributed computing
- 20:27MOAR! Distributed Systems
- 20:584G Coverage by Country
- 26:11Timeouts
- 26:31Circuit Breaker
- 29:26Retry/Timeout Gotcha
- 31:11Validate Input
- 32:18Netflix Recommendation Personal System
- 34:39Shed Load
- 36:22Google Suggest 4 Key Metrics
- 37:09Failure Injection
- 41:26Guidelines
- 42:54Monitoring & Metrics
- 43:54Sniper/Bomber Command Teams
- 47:27Further Reading
- 49:42Embrace Failure - It's Coming!
related talks
16mEngineering Failure-Resilient Systems
47mPersistent SRE Antipatterns: Pitfalls on the Road to Creating a Successful
01h 31mFive Design Patterns to build resilient Applications
45m⚡A Resilient Application in a Partially Degraded World ⚡
34mToward continuous resilience
28mBuilding Resilient DevOps Pipelines: Strategies for Handling Unexpected Failures &...
43mGetting Your System into Production and keeping It There
10mBuilding an 'Oops-Less Operation' at Scale