Efficient Ceph Performance Troubleshooting in Production Using eBPF
summary
eBPF tools trace every Ceph I/O and produce per-IO lifecycle latencies so one can see messenger, OSD queue, BlueStore substages and acting set at the client side. radostrace finds which primaries and acting OSDs cause high-tail latency; osdtrace breaks down where inside an OSD the time goes. Real fixes include identifying a rack-local NIC packet loss, upgrading disk firmware and moving rocksdb offload, and a code change from O(MN) to O(M)* for rbd sparsify (bug 69014) that yields ~60× speedup. Minimum kernel 5.8, no debug symbols required.
AI-written summary · about
chapters (14)
- 0:00Introduction to safe trace
- 1:49Motivation for life cycle latency
- 4:41OSD trace tool overview
- 8:40Production troubleshooting examples
- 13:20Running OSD trace in production
- 14:01Introduction to rados trace
- 15:27Rados trace production case
- 18:20Debugging RBD spec slowdown
- 21:35Container-based tracing demo
- 22:18Live troubleshooting demo
- 31:22Tracing librados gateway
- 33:29Kernel FS trace tool
- 34:51Future development plans
- 35:35Project resources and Q&A
related talks
39mUsing EBPF to Develop a Powerful Tool for Diagnosing Ceph Performance Issues
28mCommon Support Issues And How To Troubleshoot Them
35mMaking Ceph Fast in the Face of Failure
22mPinpoint Ceph Bottleneck Out of Cluster Behavior Mists
22mReworking Observability In Ceph
44mCeph QoS Refinements for Background Operations Using MClock
27mDebugging Linux Issues with eBPF
23mReconsidering tracing in Ceph