xprobes: Hybrid User/Kernel eBPF Probes for Cross-Layer Observability
summary
Combines uprobes and kprobes into xprobes that keep a BCC shadow stack in eBPF maps and merge kernel events with userspace entry/exit to attribute on‑CPU, off‑CPU and I/O time per function; implements a cpu‑latency xprobe that subtracts context‑switch off‑CPU time from function durations. Benchmarks trace‑all on Redis drop throughput from 450,000 → 4,000 gets/sec, showing the dominant overhead is uprobe trap/emu costs. Explores mitigations: trace fewer hot functions, compiler fentry placement, user‑space trampoline or BPF time style interfaces.
AI-written summary · about
chapters (7)
- 0:00Introduction to observability
- 1:27Latency degradation horsemen
- 3:14Introducing X probes
- 4:56Attribution logic explained
- 7:05Overhead challenges
- 9:34Current state of X probes
- 10:40Future directions and Q&A
related talks
31meBPF for Observability: The Good, the Bad, and the Ugly
32meBPF: The Next Power Tool of SREs
37mState of kprobes/trampolines batch attachment
51meBPF Warfare - Detecting Kernel & eBPF Rootkits with Tracee
36mDebugging with eBPF on Arm Platforms
54mTracing Go Programs with eBPF!
29mBuzzing Across the Cloud Native Landscape with eBPF
31mTracing, Profiling & Debugging in Production (eBPF)