Real World Debugging with eBPF
summary
Uses eBPF (implemented with bpftrace) to attach uprobes to Go binaries, compute per-goroutine timing by reading the runtime G pointer from FS-8 and struct offsets (pahole to find offsets), and avoid unsafe ureturn probes by instrumenting each return address. Finds Docker network list slowness traced to buildDetailedNetworkResources called 16 times at ~200 ms each, and shows eBPF adds <3% overhead vs GDB’s ~22× slowdown. Demonstrates goroutine-creation tracing and stack walking to root a Cilium memory leak; warns about duplicate events, missing callers and PIE build issues.
AI-written summary · about
chapters (23)
- 0:00Intro
- 2:17Reproduce & Narrow down
- 3:17Debugging Ideas
- 4:06Silver bullet: printf
- 4:45Traditional debugger: GDB
- 6:33Benchmark for GDB
- 7:30Internal tool: pprof
- 8:57eBPF: bpftrace script
- 11:14Solution to concurrency problem
- 12:01uretprobe: implementation and problems
- 13:53Solution to uretprobe problem
- 15:46Solution to TID problem
- 20:08Run bpftrace script
- 21:36Benchmark for eBPF
- 22:04Recap for Docker issue
- 26:34Traditional debugger: coredump
- 28:49eBPF: leaking goroutine tracing
- 29:39Stage 1: backtrace for leaking goroutines creators
- 35:17Stage 2: backtrace for leaking goroutines
- 40:05Recap Cilium issue
- 42:03Entrypoint: duplicate events
- 43:21backtrace: missing caller
- 45:16Happy debugging
related talks
24mDebuggers and eBPF: Bringing Debugging to Production
54mTracing Go Programs with eBPF!
36mDebugging with eBPF on Arm Platforms
36mDebugging with eBPF on Arm Platforms
27mLow-Overhead Tracing Using eBPF for Observability into Kubernetes Apps and Services
33mBuilding a Go Profiler Using Go
36mA Beginner's Guide to eBPF Programming with Go
36mObservability and performance analysis with BPF