10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You
summary
Runbook-level failures from a 1,000–2,000 node environment: CoreDNS’s autopass causes huge upstream queries that hit rate limits; IPVS + graceful termination drops UDP during rolling updates (IPVS graceful termination); an imagePullPolicy=Always daemonset on ~1,000 nodes exhausts registry quota behind three static IPs; a typo in a node-local sync selector makes kube -iam fetch the entire cluster and OOM; cronjobs create 4,000 pending pods that slow the scheduler; one host shows 16,000 zombies from a bad exec probe. These are concrete root causes and fixes.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:05Scale and outages
- 2:04Kubernetes DNS pitfalls
- 5:17IPVS and connection tracking
- 8:03Image registry bottlenecks
- 11:55API server overload
- 14:35Scheduling and resource issues
- 17:11Logging and autoscaling errors
- 19:36Cassandra node rebalancing
- 23:28Broken runtimes and zombies
- 27:35EBS I/O limitations
- 34:39Conclusion and key takeaways
related talks
35m10 Weird Ways to Blow Up Your Kubernetes
36m10 More Weird Ways to Blow Up Your Kubernetes
20m10 Common Things People Do Wrong in K8s Environments
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)
35mKubernetes, Lessons Learnt
27mRunning Kubernetes in Production A Million Ways to Crash Your Cluster
05mReading 100+ Kubernetes Post-Mortems
30mWhat we learned from reading 100+ Kubernetes Post-Mortems