How to Not Destroy Your Production Kubernetes Clusters
summary
A custom operator accidentally deletes ~20k Service objects because of a label length mismatch (63 vs 127) between cluster libs, and missing etcd snapshot forces a five‑hour manual restore; deletion also removes admission webhook objects so pods stop scheduling until webhooks are restored and a rate limiter is added. A legacy CMDB hits a 32‑bit integer overflow and its finalizers block pod lifecycle, requiring manual finalizer removal and breaking a dependency loop. A new mutating webhook rollout changes selector logic and triggers 7000 pods reconstructed in a canary.
AI-written summary · about
chapters (28)
- 0:00Intro
- 0:35Background
- 1:35Postmodern Database
- 2:02Automation
- 2:25User escalation
- 3:38Initial investigation
- 4:21Restoring service objects
- 4:49Collecting service definitions
- 5:17The impact of the incident
- 5:40The reason for the failure
- 6:23Fixing the webhooks
- 6:38Why the operator went rogue
- 7:09Kubernetes label selector package
- 7:49Test engineer accidentally created app load balancer
- 8:25What can we learn
- 9:33Paradoxical Finalizer
- 9:45Paging Storm
- 11:45Mitigation
- 12:01Kubernetes Platform
- 12:28Manual Operations
- 12:49Lessons Learned
- 13:59User Complaints
- 14:21Monitoring Dashboard
- 14:52Victim Cluster
- 15:22Security Context Change
- 17:06Learnings
- 18:44Recap
- 19:13Key takeaways
related talks
36m10 More Weird Ways to Blow Up Your Kubernetes
27mRunning Kubernetes in Production A Million Ways to Crash Your Cluster
31mHow the OOM-Killer Deleted My Namespace, and Other Kubernetes Tales
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)
09mLightning Talk: How to Avoid a Kubernetes Doom Loop
20mYou can't have a cluster*!$& without a cluster!
35m10 Weird Ways to Blow Up Your Kubernetes
36m101 Ways to “Break and Recover” Kubernetes Cluster