What We Learned from Reading 100+ Kubernetes Post-Mortems
Talkat KubeCon + CloudNativeCon North America 2021 · on Friday, 2021-10-15· at 17:25· Room 403 AB + Online·
summary
Team reviews 100+ post-mortems and extracts concrete failures: a CronJob default concurrency spawns 4,000 pods and kills the API server, a wildcard ingress routes all traffic to one service taking down the cluster, and missing CPU/memory limits cause OOMs in production. Recommended fixes are explicit organizational policies, automated checks in CI, and centralized policy management; tools demonstrated are Gatekeeper, Conftest, Datree to enforce, shift-left, and monitor configuration guardrails.
AI-written summary · about
chapters (21)
- 0:00Intro
- 1:25About Datree
- 3:36The mistake game show
- 4:03What is the mistake
- 4:36The concurrency policy
- 5:00What happened to Target
- 5:56Never trust default configuration
- 6:25Concurrency policy
- 6:58YAML structure
- 7:13Wrong YAML placement
- 8:37ingress resource
- 9:45star
- 10:36delegate the knowledge
- 11:05pod
- 17:23how can you prevent it
- 20:51Integration
- 22:35Control Review Monitor
- 23:54Overview
- 24:41Gatekeeper
- 27:58Datree
- 31:39Outro
related talks
05mReading 100+ Kubernetes Post-Mortems
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)
42mMastering Kubernetes: Challenges, Strategies, and Best Practices
41mKubernetes the Very Hard Way
45mWhat I Learnt Fixing 50+ Broken Kubernetes Clusters
32mSurviving From Endless Issues Coming From 7K+ Kubernetes Clusters
18mKubernetes and the Answer is… 42!
31mThe Myth of the Monocluster