“Using async-profiler to Find Tail Latency Issues”
summary
Shows how async-profiler-style sampling plus a tiny per-task timestamp reveals where Tokio tasks stall: a thread-local write when a task is polled (about ten cycles) lets samples be attributed to long Tokio task timestamping periods. Finds real cases of repeated allocator churn—malloc freeing to the OS and reacquiring—causing reproducible 6 ms pauses in production. Tooling (open-source pcatch crate on GitHub) writes data when tokio stable is enabled and pinpoints the offending code paths so the allocator behavior can be fixed.
AI-written summary · about
chapters (7)
- 0:00Rust at AWS
- 0:53Understanding tail latency
- 1:38CPU-bound issues in Tokio
- 2:20Limitations of current tools
- 3:03Introducing PollCatch
- 3:47Real-world debugging case
- 4:31Tool availability and usage
related talks
17mZero-Copy Chaos. When everything you knew about tokio...
22mUsing Tokio for CPU-Bound Tasks (Works Really Well)
33mStorage-intensive apps on tokio: when it's time to reach for io_uring
35mScalable Tokio timer for multi-core machine
38mLess Tokio is More Tokio: Strategies for Accelerating a Rust Native Search Database
13mLightning Talk: dial9: WTF is Tokio actually doing?
30mDay2 02 Sean Klein
04mThe Power of Profiling