What we learned from reading 100+ Kubernetes Post-Mortems
summary
Team reads 100+ post-mortems and finds recurring root causes: bad cronjob concurrency settings that spawn thousands of pods (Target, Zalando), missing memory limits that trigger cluster OOMs (BlueMatador), simple YAML/schema mistakes that delete namespaces (Skyscanner), and widespread knowledge gaps. Recommended fixes use kubeconform and yq for syntax/schema checks, run validation in CI, enforce policies gradually, and use centralized policy tooling — built an open source the tree CLI plus a policy manager that validates locally and in CI and propagates changes across pipelines.
AI-written summary · about
chapters (9)
- 0:00Introduction and meetup story
- 7:26Learning from failure stories
- 11:00Kubernetes misconfiguration game
- 16:17Policy enforcement best practices
- 20:49Validating syntax and schemas
- 23:15Defining and enforcing policies
- 24:23Centralized policy management
- 26:29Introducing the Tree tool
- 29:37Conclusion and Q&A
related talks
05mReading 100+ Kubernetes Post-Mortems
34mReading 100+ Kubernetes Post-Mortems - lessons learned
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)
29mKubernetes Failure Stories and How to Crash Your Clusters
37m10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise You
35mKubernetes, Lessons Learnt
30mLearning From Failure: Post Mortems
42mMastering Kubernetes: Challenges, Strategies, and Best Practices