Reading 100+ Kubernetes Post-Mortems - lessons learned
summary
Hundreds of postmortems (100+) boil down to concrete mistakes: default cronjob settings and misplaced YAML cause 4,300+ restarting pods in one incident, wildcard ingress routes all traffic to one pod, and missing resource limits lets third‑party containers hog memory. The tooling story is specific: Gatekeeper (OPA admission webhook) protects the cluster at apply time, conftest in CI catches issues earlier, and a Datree CLI with predefined rules and centralized policy management claims to prevent misconfigurations by running checks locally, in CI, or as admission controls.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:17Talk overview
- 3:23Motivation
- 5:51100+ postmortems lessons
- 6:55CronJob concurrency misconfig
- 10:28CronJob YAML errors
- 14:21Ingress wildcard host issue
- 18:58Missing Pod resource limits
- 21:43Automating misconfig prevention
- 26:41Kubernetes policy tools
- 32:20Key takeaways
- 33:03Q&A
related talks
05mReading 100+ Kubernetes Post-Mortems
30mWhat we learned from reading 100+ Kubernetes Post-Mortems
36mKubernetes Confessions: Real Mistakes We’ve All Made (and What They Teach Us)
42mMastering Kubernetes: Challenges, Strategies, and Best Practices
35mKubernetes, Lessons Learnt
29mKubernetes Failure Stories and How to Crash Your Clusters
33mKubernetes Security: Horror Stories to Help Secure Your Cluster
20m10 Common Things People Do Wrong in K8s Environments