The Unconspicuous Role of Conntrack in Kubernetes Networking
summary
On Sept 4, 2025 a 100x DNS request spike and Istio’s aggressive service entries refresh (every 30s) combined with Envoy reusing one socket causes all pods to share a single source port; conntrack treats UDP with a 120s bidirectional timeout so entries never expire, pinning kube-proxy’s IPTables load‑balancing to one CoreDNS endpoint. With ~90 CoreDNS pods this funnels cluster volume to one pod, triggering OOM kills and widespread DNS 500s; upgrading to kube‑proxy with the conntrack reconciler (k8s 1.32) or changing Istio/Envoy behavior resolves it.
AI-written summary · about
chapters (10)
- 0:00Introduction to incident
- 2:00Kubernetes pod architecture
- 4:06Name resolution mechanics
- 7:55Network traffic path
- 14:42Load balancing with kube-proxy
- 16:52Detailed ingress path
- 20:06Initial incident investigation
- 20:54The role of Istio
- 23:50Conntrack and stale entries
- 28:08Conclusion and remediation
related talks
38mLogs Told Us It Was DNS, It Looked like DNS, It Had to Be DNS, It Wasn't DNS
35mKubernetes, Lessons Learnt
32mUnderstand and Troubleshoot the “Magic” of Kubernetes Networking - Minhan Xia & Rohit Ramkumar
28mHow to Break your Kubernetes Cluster with Networking
06mLightning Talk: Is Your Kubernetes Cluster's DNS Working?
31mThirty seconds of silence
43mKubeCon: A Story of Two Service Discoveries: Bringing Legacy to Kubernetes
51mKubernetes SIG-Network: Intro and Deep-Dive