Kubernetes Failure Stories and How to Crash Your Clusters
summary
Zalando runs ~118 clusters across ~380 AWS accounts for ~900–1,000 developers and presents eight concrete failures: a shared ALB + Skipper OOM that knocked healthy backends to zero; a mis-indented cronjob that exploded pods to ~3,000; CoreDNS OOM requiring memory bumps to 4GB and node-local caching; an etcd/API timeout traced to T2 instance packet drops; an upgrade-path bug fixed by adding in-place upgrade tests; a kubelet memory leak to ~9GB; and CPU throttling that backed up a secrets injector to 58k queued items.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:45Checkout payment failure
- 5:54Product detail page errors
- 7:40CoreDNS OOM crash
- 10:08kubectl delete human error
- 11:13API server latency spikes
- 14:23Cluster upgrade failure
- 17:09kubelet memory leak
- 18:55Secret injection CPU throttling
- 22:26Failure causes and lessons
- 27:17Q&A
related talks
36m101 Ways to Crash Your Cluster
35mKubernetes, Lessons Learnt
30mWhat we learned from reading 100+ Kubernetes Post-Mortems
34mReading 100+ Kubernetes Post-Mortems - lessons learned
20mYou can't have a cluster*!$& without a cluster!
32mWho Killed My Pod? #Whodunit
39m15 KUBERNETES FAILURE POINTS YOU SHOULD WATCH
34mPanel: Improving and Managing Kubernetes at Scale