Restarts and rollbacks don't fix everything
summary
Claims concrete results: deployment failures drop from 1 in 200 to 1 in 10,000 by using deployment docs, canaries, an operations bar raiser, and automated rollbacks with a 5/5/5 deploy evaluate rollback target. Practical tools include rate-limited control-plane APIs, pair-reviewed manual changes, warm pools and IP caching to survive S3 or AZ blips, and shifting control-plane state from relational DBs to DynamoDB/QLDB. Examples include accidental bulk deletes and replication storms and a demoable remediation model that runs fleet-wide debugging commands in parallel.
AI-written summary · about
chapters (15)
- 0:00Intro
- 1:08Anurags background
- 4:28Why do systems fail
- 6:00Deployment issues
- 10:22Auto roll back
- 12:14Operator error
- 13:38Manual changes
- 16:09Black box components
- 18:36Control your own fate
- 28:35Thank you
- 31:04Operations vs Leadership
- 33:01Deployment vs Systems
- 35:04System failures
- 36:27Shoreline
- 38:38Advice for young DevOpsers
related talks
02mOpening Remarks
53mDeliver Results, Not Just Releases: Control & Observability in CD
16mEngineering Failure-Resilient Systems
39mOops!… We F*ed It Up Again: The Art of Staying Calm When Systems Crash
40mThe Art of Embracing Failure at Scale
35mThe Virtuous Cycle: Getting Good Things out of Bad Failures
43mTo make mistakes is human. To auto-deploy it is DevOps.
26mSelf Healing Systems