Troubles and Tidbits from Datadogs eBPF journey
summary
Datadog describes concrete failures using eBPF in production: building Cloud Security Management with kprobes/tracepoints and Cilium hooks exposes blind spots like missing 32‑bit compatibility hooks, kprobe max_active and interrupt reentry dropping probes, and module reloads losing instrumentation. User pointers require copying from kernel, ring buffers drop events under pressure, interpreters hide executed scripts, TC filter races break coexistence, OOM can kill agents and lose kprobes, and Cilium faces LRU map evictions and sk_mark bit‑shuffle corrupting identity; a reuseport + bpf_sk_assign kernel patch finally fixes a broken load‑balancing experiment.
AI-written summary · about
chapters (7)
- 0:00Introduction to Datadog and eBPF
- 1:26Cloud Security Management with eBPF
- 3:55Hook points pitfalls
- 10:15Data collection challenges
- 12:44Environment and kernel issues
- 17:17Cilium networking challenges
- 27:56Q&A session
related talks
38mUnderstanding and Troubleshooting the eBPF Datapath in Cilium
40mPanel Discussion: eBPF: A New Era in Cloud Infrastructure Tools
32mExploring the Power of eBPF for System Observability
31meBPF for Observability: The Good, the Bad, and the Ugly
22mTales from an eBPF Program’s Murder Mystery
42mCommunity Driven Breakout Session: eBPF and Auto-Instrumentation- 40mPanel Discussion: eBPF: A New Era in Cloud Infrastructure Tools
31mBridging Dev and Ops with eBPF: Extending Observability Upwards and Downwards