How Security Requirements Turned Us into Accidental Chaos Engineers
summary
A cron job called Lifespan Management enforces max age on AWS VMs by terminating ASG instances and letting replacements boot; it scales terminations roughly 10x, exposed that health checks can lie, revealed two different EC2 terminate APIs (one a hammer), and turned graceful shutdown into a black hole unless changed. The team builds a terminator service, adds two-touch termination, automates retries, reduces noisy JIRA tickets, and achieves broad adoption while helping with Spectre/Meltdown kernel rollouts.
AI-written summary · about
chapters (29)
- 0:00Intro
- 1:12Terminology
- 1:22Instance Age
- 2:23What's wrong with old instances?
- 3:58CVE Patch Released
- 5:11Components
- 5:28What is an auto-scaling group?
- 5:45What is a terminator?
- 6:26Lifespan Manager
- 6:51Terminate First vs. Launch First
- 8:24Breaking it up with labels
- 8:48A Year Long Journey
- 9:21Things got Chaotic
- 10:40How did this happen?
- 11:32How NOT to health check
- 12:06Explicit Expectations
- 12:33Terminator Recap
- 13:57RIP Kubernetes Workers
- 14:49Heartbeat Options
- 16:35Two Touch Termination
- 17:03The Blackhole Scenario
- 20:43Turning it on
- 21:14Who does what?
- 21:43False Alarm
- 22:18Self-service Meltdown
- 24:16Death by a thousand JIRA tickets
- 25:34The Last Mile
- 26:55Summary
- 28:02Takeaway
related talks
27mHow Our Security Requirements Turned Us into Accidental Chaos Engineers
57mChaos Engineering Primer
42mChaos Engineering: Why Breaking Things Should be Practiced
44mDevSecOps: Balancing Agility, Security & Stability in a Continuous Software Business
17mAdvance securely: Safely migrating to
33mImprove resilience with automated chaos engineering
33mAutomating Cloud Security and Incident Response (DevSecOps)
53mLessons Learned: A Retrospective on Application Security Failures