Surviving Operational 'Chernobyl'
summary
Covers concrete incident responses and outcomes including a lifesaving service that handles 1.5 million eCalls over two years and loses about 85 potential rescues per hour of outage, several Jenkins outages that force manual deployments and then a dedicated Jenkins instance, an emergency disable 2FA config change to restore logins in 45 minutes, terraform drift that wipes configs, Log4j emergency upgrades rolled out worldwide in six hours, backups taken before suspected compromise and postmortems that translate hotfixes into permanent fixes.
AI-written summary · about
chapters (17)
- 0:00Introduction to operational Chernobyl
- 2:12DevOps plugin approach explained
- 4:00Life-saving operations in production
- 6:55Preparing for critical production issues
- 10:01Handling critical incidents on production
- 11:48Post-incident analysis and fixes
- 13:55Real production failure cases
- 23:27Being first in cloud pitfalls
- 25:15Jenkins outage nightmare
- 31:20Team communication through escape rooms
- 32:02Log4j zero-day exploit response
- 34:54Floating code and visibility issues
- 36:49Manual account creation risks
- 38:49Cloud message flow delays
- 40:37User-reported vs production issues
- 43:08Ongoing production issue resolution
- 45:54Summary and final advice
related talks
42mSurviving operational "Chernobyl"
54mSurviving operational "Chernobyl"
47mHow did I Help to Save the Lives of More Than 1.5 Million People?
30mWhat Software Engineers Can Learn From the Chernobyl Disaster
16mEngineering Failure-Resilient Systems
30mThe last in the line: a talk about DevOps and Disasters
55mDevOps @Scale (Greek Tragedy in 3 Acts)
23mDiving through RCAs a quest for reliability