Surviving From Endless Issues Coming From 7K+ Kubernetes Clusters
summary
Kakao reports running 7K+ clusters (100K+ nodes) and a huge on-call load, and describes building an open source CLI called detect that collects Kubernetes API, Prometheus, SSH and infra data, runs >50 rule detectors (public repo has ~6), outputs JSON/HTML reports and can auto-remediate or warn. The team quantifies surprises (thousands of unused clusters, pods restarting 12,000 times, expired TLS certs), shows rule implementation examples, and says the tool reduces repetitive human debugging.
AI-written summary · about
chapters (19)
- 0:00Surviving Kubernetes at scale
- 2:55Why many small clusters
- 4:23Unexpected challenges at 7K+
- 5:06Unused clusters waste resources
- 6:35Diagnosing recurring cluster issues
- 7:39Common developer mistakes in Kubernetes operations
- 8:44Need automated detection
- 10:12Introducing Detect
- 11:16Live demo of Detect
- 14:10Custom rules for Detect
- 21:56Conclusion and open-source
- 22:17Q&A: Detection rules
- 23:41Q&A: Detect deployment
- 25:29Q&A: Upgrades at scale
- 26:40Q&A: Detect permissions
- 27:43Q&A: Dev vs cluster count
- 28:51Q&A: Detect execution rights
- 29:57Q&A: Cluster infrastructure
- 30:40Q&A: Handling outages
related talks
06mSponsored Keynote: What I Learned Running 10,000+ Kubernetes Clusters
33mAm I Using It Right? Checking Best Practices on Live Kubernetes Clusters
29mProduction Cluster Monitoring and Remediation for High Reliability
45mWhat I Learnt Fixing 50+ Broken Kubernetes Clusters
18mK8s Troubleshooting Demystified: Five Best Practices
34mReading 100+ Kubernetes Post-Mortems - lessons learned
41mKubernetes the Very Hard Way
12mKeynote: From Outage To Observability: Lessons From a Kubernetes...