Surviving operational "Chernobyl"
summary
Team runs a DevOps competence center supporting a life‑critical eCall system on AWS that sees 1.5 million eCalls over two years (missing ~100 calls/hour risks lives). Fourteen real incidents include frontend removing the “end call” button, 2FA outage temporarily disabled, primary and failover firewall/power failures, pagination causing 2.5‑minute pages, Terraform runs wiping configs, Jenkins outages on shared instances, Docker pods rebooting due to test servers hitting production, and a Log4J patch rolled out in six hours; final tally 2 vs 12 (DevOps vs other teams).
AI-written summary · about
chapters (22)
- 0:00Welcome and introduction
- 2:35DevOps sub plug-in approach
- 6:33Documentation and emergency tests
- 11:14Missing the SLA
- 12:39Free calls issue
- 14:27Two-factor authentication failure
- 16:54Network infrastructure failure
- 18:39Power outage in data center
- 19:22Lack of pagination in refunds list
- 21:06Cloud Terraform script disaster
- 22:52Jenkins deployment issues
- 24:41Missing database index
- 26:09Azure message flow problems
- 28:01Log4j zero-day exploit
- 30:07Deployment calendar importance
- 31:14Cross-team issue visibility
- 32:18Message flow delay and limits
- 33:21Manual database changes
- 34:02Shopping incident and network issues
- 35:25Docker image rebooting problem
- 37:35Final score and summary
- 38:17Q&A and closing remarks
related talks
54mSurviving operational "Chernobyl"
48mSurviving Operational 'Chernobyl'
47mHow did I Help to Save the Lives of More Than 1.5 Million People?
55mDevOps @Scale (Greek Tragedy in 3 Acts)
48mDevops@Scale
31mDevOps in the Cloud: Case Studies of Amazon.com teams and their resilient architectures
30mThe last in the line: a talk about DevOps and Disasters
36mDevOpsSup - Extended DevOps approach