Building For Gracious Failure
summary
Systems need concrete practices not platitudes: Bugsnag exposes spikes (231,000 vs ~2,000 and ~10,000 errors), SignalFx and custom metrics show jobs started ≈ jobs failed, a Go job processor and a Business Profile service that migrated 9 million data points produces parse errors. The team changes parsing to return null for bad fields and return what we can, accepts partial JSON updates with validation to accept what we can, records metrics for failures, and trusts carefully between services to avoid cascading 500s.
AI-written summary · about
chapters (5)
- 0:00Start
- 2:15We Can't Fix What We Can't See
- 12:05Return What We Can
- 19:44Trust Carefully
- 25:35Questions
related talks
51mDesigning for Failure: Build Resilient Services
25mFail Faster: Adding Circuit Breakers to your APIs
52mSurviving in a Microservices Team
42mArchitecting for Failures in Microservices
33mArrested Development
49mScaling Reliability: So You Want to Add a 9
45mMonitor, Fix and Deploy (on Friday)
57mLearning to Love your Non-functionals