Understand and Troubleshoot the “Magic” of Kubernetes Networking - Minhan Xia & Rohit Ramkumar
summary
Real cases from GKE show concrete failures and fixes: UDP DNS to a ClusterIP repeatedly blackholes because conntrack sends packets to a removed pod, fixed by having kube-proxy flush stale conntrack entries when endpoints change; host VM OOM kills networkd and a kernel-resetting sysctl breaks forwarding until a COS kernel patch lands; hostPort + Service rules collide because a broad iptables rule DNATs port 443 first — an iptables precedence problem; debug uses iptables-save, conntrack dumps, tcpdump on the pod veth pair, and strace to trace where packets stop.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 0:58Black Hole
- 2:51Contract
- 4:03NetFilter
- 5:21Blackhole
- 6:06What we learn from this
- 8:04Another case for a black hole
- 10:03What happened
- 11:03What we learn
- 12:00A wormhole
- 13:17IPtables
- 15:40Things we learned
- 16:28Best practices
- 17:00IP tables
- 17:22Cube Services
- 24:23Conclusion
- 25:03What about IPBS
- 26:43Other tools we recommend
related talks
35mKubernetes, Lessons Learnt
28mHow to Break your Kubernetes Cluster with Networking
20mKubernetes Networking 101
34mIt's Not Always the Network (But Here's How to Prove It): Kubernetes Packet...
30mThe Unconspicuous Role of Conntrack in Kubernetes Networking
40mAttacking Kubernetes Clusters Through Your Network Plumbing
27mDeconstructing K8s Networking for Knowledge and Profit
51mHow Kubernetes Networking Really Works: A Packet’s Journey Across Pods and Nodes