Effective Disaster Recovery: The Day We Deleted Production
summary
A one-line config change in an Argo CD app-of-apps setup where auto-sync plus prune ran against a name-collision one-line typo caused prune deleted a production cluster; most PVs were Retain so recovery focuses on manual re-creating Application objects and reattaching volumes reconnect PVs instead of redeploy rather than replaying S3 data. Incident response: hit the “big red button” to stop syncs, follow a deployment checklist, scale ingress, let task backlog drain, and restore services — total recovery under six hours. Postmortem changes: commit generated YAML, a collision detector, and argo cd prune=false annotation at namespace level.
AI-written summary · about
chapters (17)
- 0:00Start
- 1:29Rick Spencer
- 1:48Wojciech Kocjan
- 3:34InfluxDB Cloud, stateful, GitOps & CI/CD
- 5:44How it started - a PR gets merged
- 6:51How ArgoCD works
- 7:50What ArgoCD did
- 8:57Good Morning!
- 9:32The Damage is Done
- 11:44The Rebuild, 11:17 - 15:26
- 13:52Return to Service, 15:26 - 16:04
- 15:16All Clear, 16:04 on
- 16:21Summary of recovery
- 18:54First instinct - revert the change
- 23:05Ensure such change wouldn't get merged
- 24:57Improve ArgoCD configuration
- 26:33Improve processes around outages
related talks
05mWe accidentally deleted production, but we made it not matter all that much
24mWhoops! How Not to Accidentally Delete Everything.
14mDisaster Recovery preparedness
30mIs Your Kubernetes Disaster Recovery Actually Ready?
46mA journey to disaster
32mSchroedingers Backup
25mRUD: Rapid Unscheduled Deletion of Critical Infrastructure Workloads
33mNavigating Disaster Recovery in Kubernetes and Crossplane: Strategies and Tools for Managing Stateful Workloads with GitOps for Platform Teams