The Meat of It
summary
Incident reports surface concrete failures: an accidental commit enabling 100 tracing caused a retry storm that lowered observability and is fixed by returning tracing; canary ergonomics matter — a purple canary queued behind yellow canary and later picked up an unintended commit; a spreadsheet change switches SQL to GraphQL and generates 500x more graphql calls, which floods downstream services and fills a database disk. Lessons focus on actual numbers, configuration paths, queues, and what was built and why it failed.
AI-written summary · about
chapters (11)
- 0:00Intro
- 2:25The Meat of It
- 3:56Resilience Engineering
- 4:39Reification
- 7:33Oversimplification
- 13:06Dependency Graph
- 14:29Customer Impact Window
- 18:53Website Overwhelm
- 23:08Database Overwhelm
- 26:16Stella Report
- 27:22Conclusion
related talks
01h 03mWhispers in the Chaos: Monitoring Weak Signals
15mWhy I Prefer Good Testing Over Excellent Testing
30mUsing incidents to level-up your teams
34mIncidents are new normal
23mHow Tracing Uncovers Half-truths in Slack’s CI Infrastructure
29mObservability for Developers: How to Get From Here to There
32mBuilding for Gracious Failure
28m"What Happened in Production?! Instrumenting with OpenTelemetry"