Who Dropped the Packet? Solving K8s Network Mysteries in Real-Time
summary
Operator loads eBPF programs in the kernel, exports metadata (not full packets) to a flow pipeline that enriches records with Kubernetes API info and stores in Loki/Kafka, and surfaces a traffic table, drop view and topology. Demo breaks an egress rule and immediately shows an connection tracking table full explicit drop despite 7,000+ OVN rules in a 3‑node test; root cause hunting drops from weeks to ~30 minutes. Sampling controls visibility (sampling=1) and measured overhead is tiny (0.08 CPU across six nodes).
AI-written summary · about
chapters (12)
- 0:00Packet drops in Kubernetes
- 1:24Kubernetes networking complexity
- 3:50Traditional tools limitations
- 4:53eBPF kernel visibility
- 7:00Network Observability Operator
- 11:58Intermittent timeouts case study
- 14:05Packet drop detection demo
- 25:24eBPF sampling
- 28:13eBPF architecture details
- 31:22Efficiency and takeaways
- 32:06Resources and next steps
- 34:51Q&A
related talks
34mIt's Not Always the Network (But Here's How to Prove It): Kubernetes Packet...
35mFollowing a packet journey through the networking stack in OpenShift
24mDeep Network Traffic Observability with Pktvisor and Prometheus
29m“Are These Things Talking to Each Other?” - Observing Kubernetes Networking...
27mStep By Step Kubernetes Observability with eBPF
27mBeyond printf & tcpdump: Debugging Kubernetes Networking with eBPF
51mHow Kubernetes Networking Really Works: A Packet’s Journey Across Pods and Nodes
34mLife of a Packet [I]