Try Catch Blocks for your Distributed System
summary
Argues for concrete practices: run design and code reviews, validate rolling deployments for dual-version compatibility, and exercise canaries that are canary that represents real customer traffic. Instrument per-dependency metrics — measure latency, throughput, return-code rates and exceptions separately — and set layered alarms (short high-threshold for spikes plus long-window aggregates). Limit retries, use backoff and jitter to avoid retry storms, prefer local caches, simulate failure modes or run game days, then post-incident add diagnostics and backlog fixes.
AI-written summary · about
chapters (16)
- 0:00Intro
- 2:38Design Reviews
- 8:20Integration Testing
- 13:41Metrics
- 17:20Canaries
- 18:46Turbulence
- 20:26More metrics
- 22:01Alarms
- 24:12Handling failures
- 24:59Retries
- 28:15Loadshedding
- 29:02Caching
- 30:07Learning from failures
- 33:08Learning your system
- 34:12Handling and prevention
- 35:26Why
related talks
52mPerformance & Stability Anti patterns
18mChaos-Driven Development: TDD for Distributed Systems
01h 02mFive things every developer should know about building mission-critical systems
29mPractical distributed systems: Building for the real world with chaos engineering
46mPracticalities of Productionizing Distributed Systems
43mGetting Your System into Production and keeping It There
35mDesign patterns for application resiliency
31mTesting in Production