Debugging Linux Issues with eBPF
summary
A Kafka cluster sees a 10x CPU increase after a Debian/kernel upgrade; perf and flamegraphs show spinlock contention and many timer callbacks. Using eBPF/BCC and perf to count stacks and timers reveals a 12x increase in TCP write timers caused by missing TCP segmentation offload on the VLAN interface, which forces software segmentation and extra locks. Enabling offload and restarting Kafka restores throughput and cuts system CPU usage back to normal.
AI-written summary · about
chapters (9)
- 0:00Context and infrastructure
- 2:28OS upgrade and Kafka issue
- 6:27Initial debugging steps
- 7:38Perf and root cause analysis
- 12:06Flame graph investigation
- 13:57BCC tools for deep analysis
- 16:53Network offload discovery
- 21:23Resolution and fix validation
- 22:56Key takeaways and BPF
related talks
38mThe Next Linux Superpower: eBPF Primer
27mAnalyzing Postgres performance problems using perf and eBPF
27mLow-Overhead Tracing Using eBPF for Observability into Kubernetes Apps and Services
38mEfficient Ceph Performance Troubleshooting in Production Using eBPF
34mTroubles and Tidbits from Datadogs eBPF journey
36mObservability and performance analysis with BPF
32meBPF: BPF kernel infrastructure
38mLinux Kernel Tracing Using eBPF