Oops!… We F*ed It Up Again: The Art of Staying Calm When Systems Crash
summary
Describes concrete changes: replace noisy log scans with realtime alerts and SLO-driven monitoring, add a one‑page incident wiki and playbooks, shift tests left to canary and service integration tests and use local API mocks (nock), enforce feature flags and prefer reboot rollback revert restore over risky hotfixes. Results are surprising: target was 99.95% and <3 incidents in six months but the system hits >4 nines availability, zero incidents in a year, 96% automated detection and an MTTR of 20 minutes.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 2:36Who we are
- 4:46Root causes of crashes
- 5:28Setting KPIs
- 6:29Discoverability with shift left/right
- 8:33Shift right for monitoring
- 10:19Tools and team knowledge
- 11:45Preparing juniors for on-call
- 14:15Playbooks for incidents
- 16:00From fixing to stopping the bleed
- 19:52The four Rs
- 23:50Forward fixes risks
- 24:33Maintaining improvements
- 25:55False alerts danger
- 29:48Retros and postmortems
- 33:20Prioritizing action items
- 35:04Results after four Rs
- 36:08Healthy on-call culture
- 38:17Closing thoughts
related talks
15mMaintaining reliable systems
34mIncidents are new normal
26mMission Critical Mindset Building a Culture of Resilience in DevOps Teams
15mIncident Ready: Chaos Engineer Your Incident Response
45mWhat to do when complex systems fail: A guide to chaos management
41mLessons from Mission-Critical Systems in Fast-Moving Industries
43mBetter Software Development Systems through Stress
31mThe Art of Turning Incidents into Opportunities