Using EBPF to Develop a Powerful Tool for Diagnosing Ceph Performance Issues
summary
Built an eBPF/uprobe tool that measures per-request Ceph OSD lifecycles across messenger, OSD and BlueStore (receive, dispatch, queue, osd, bstore alloc/data/kv, total) and passes addresses via a DWARF-based address parser so offsets aren’t hardcoded. Demonstrates that 25% packet loss pushes receive latency from ~400 µs to 8–15 ms, and 4K writes bump BlueStore allocation from ~100 µs to several ms. Overhead per probe is ~1–3 µs; demo runs on three OSDs and will be open sourced.
AI-written summary · about
chapters (9)
- 0:00Introduction to eBPF and Ceph
- 1:07OSD operation lifecycle
- 5:08Limitations of current methods
- 8:24Introduction to u-probe
- 15:10eBPF technology overview
- 23:10Debug symbols and address resolution
- 23:31Demo of OSD Trace tool
- 32:30Future work and improvements
- 33:33Q&A session
related talks
38mEfficient Ceph Performance Troubleshooting in Production Using eBPF
40mPanel Discussion: eBPF: A New Era in Cloud Infrastructure Tools
31meBPF for Observability: The Good, the Bad, and the Ugly
37meBPF: Revolutionizing Observability and Telemetry for Modern Applications
43meBPF Superpowers for SRE
34mTroubles and Tidbits from Datadogs eBPF journey
32meBPF: The Next Power Tool of SREs
31mBridging Dev and Ops with eBPF: Extending Observability Upwards and Downwards