How the OOM-Killer Deleted My Namespace, and Other Kubernetes Tales

on YouTube ↗31m

" summary

Datadog runs dozens of clusters (some ~4,000 nodes) and shares three root-cause stories: a refactored Helm chart moved the Namespace out of the release so helm upgrade removes resources and a metrics-server APIService discovery failure (caused by a certificate reload/feature-gate mismatch) leaves the namespace controller unable to delete contents for four days; a CNI ptp delete hangs due to a netlink syscall bug triggered by a kernel jump (Ubuntu moved from 4.15 to 5.4) so netlink bug on kernel 4.20 blocks kubelet and container runtime checks; and a newly deployed node-group controller issued massive parallel lists that hit etcd because lists bypass cache (resourceVersion unset), accumulating ~2 days of query time and overloading kube-apiserver.

AI-written summary · about

" chapters (33)

1. 0:00Intro
2. 0:35Datadog
3. 1:55Symptoms
4. 2:14Investigation
5. 2:51Deletion call, 4d before Audit logs for the namespace
6. 3:45Spinnaker deploys (v1)
7. 4:17Helm 3 deploys (v2)
8. 4:33Big difference
9. 5:12What happened?
10. 6:47Namespace Controller logs Virtual
11. 8:10Events so far
12. 8:59Metrics-server setup
13. 9:41Metrics-server deployment
14. 10:27Full chain of events
15. 10:55Key take-away Apiservice extensions are great but can impact your cluster
16. 12:00Context
17. 12:49Runtime is down?
18. 14:10CNI status
19. 14:49Containerd goroutine dump Blocked goroutines?
20. 15:27Seems CNI related
21. 15:58What about Delete?
22. 16:23CNI plugin
23. 17:05The root cause
24. 19:19What we know
25. 19:38Apiserver requests
26. 22:13Illustration
27. 23:01What about label filters?
28. 24:01Informers instead of List How do informers work?
29. 26:14Back to the incident
30. 27:25Nodegroup controller?
31. 28:12How did it work?
32. 29:07What we learned
33. 29:46Conclusion

" related talks

• 21mHow to Not Destroy Your Production Kubernetes ClustersQian Ding · SREcon22 Asia/Pacific
• 36m101 Ways to “Break and Recover” Kubernetes ClusterSuresh Visvanathan, Nandhakumar Venkatachalam · KubeCon + CloudNativeCon Europe 2018
• 34mReading 100+ Kubernetes Post-Mortems - lessons learnedNoaa Barki, Shimon Tolts · Conf42 Site Reliability Engineering 2021
• 31mGhosts in the Runtime: Who Ate My Capabilities & Other MysteriesLaurent Bernaille, Eric Mountain · KubeCon + CloudNativeCon Europe 2021
• 30mWhat we learned from reading 100+ Kubernetes Post-MortemsNoaa Barki · Berlin Buzzwords 2022
• 37m10 Ways to Shoot Yourself in the Foot with Kubernetes, #9 Will Surprise YouLaurent Bernaille · KubeCon + CloudNativeCon Europe 2019
• 35mKubernetes, Lessons LearntKobi Biton · SREday London 2022
• 41mKubernetes the Very Hard WayLaurent Bernaille · LISA19