Reconsidering tracing in Ceph
summary
Hypothesis: replacing bufferlist's std::list with vector or deque changes performance. Uses LTTng tracepoints capturing instructions, CPU cycles, L1/L2/TLB misses and a microbenchmark of 1000 lists × 1000 buffers with turbo and SMT disabled. Results: vector wins for append when no reallocation, deque often has the best push_front/push_back performance, list shows consistent instruction counts but more L1 misses, larger buffer sizes increase cache misses. Attempts to run full Ceph used unit tests (daemon crashes), logging via tracepoints removes locks but shows string work dominates, and Trace Compass reveals RADOS critical-path wake/block inefficiencies.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:22Outline
- 1:05Hypothesis
- 3:08Analysis
- 4:22Micro benchmark
- 5:02Results
- 11:10Push front
- 12:25Unit tests
- 13:37Logging
- 14:50Trace Compass
- 15:36Trace Compass Demo
- 15:49Trace Compass Overview
- 22:23Outro
related talks
38mEfficient Ceph Performance Troubleshooting in Production Using eBPF
52mTracking Down What Happened: A Comparison of Tracing Solutions
39mUsing EBPF to Develop a Powerful Tool for Diagnosing Ceph Performance Issues
38mLTTng: Beyond Ring-Buffer Based Tracing
54mUnderstanding Performance w/DTrace (While The Customer Yells)...
41mAnalyzing Your System with Tracing Libraries
35mLinux Tracing updates
48mCyclictest latency, find out why with tracing