Optimizing incident response
summary
A non-production Kubernetes cluster experiment injects terminations of child pods parent pods and the node while streaming client-facing logs; the run documents an observed logs webpage drift from cluster pod logs, a few small bugs, and both a safe delete command and a catastrophic unsafe delete. Weeks later a production incident reproduces the parent-pod failure and the pre-recorded safe delete is executed, avoiding downstream outages and measurably reducing mean time to recovery.
AI-written summary · about
chapters (9)
- 0:00Incident response optimization
- 1:04Introduction
- 2:10Chaos engineering basics
- 3:59Chaos engineering impact
- 4:41Kubernetes chaos setup
- 6:05Incident resolution with findings
- 7:30Lessons from chaos engineering
- 8:53Chaos engineering in production
- 9:34Experiment and break systems
related talks
12mKeynote: From Outage To Observability: Lessons From a Kubernetes...
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)
34mReading 100+ Kubernetes Post-Mortems - lessons learned
20mYou can't have a cluster*!$& without a cluster!
34mWhat Happens When Something Goes Wrong? On Kubernetes Reliability
33mWhat We Learned from Reading 100+ Kubernetes Post-Mortems
36m101 Ways to Crash Your Cluster
29mKubernetes Failure Stories and How to Crash Your Clusters