Profiling Request Latency with Critical Path Analysis
summary
Uses the Go execution tracer and a backward critical path analysis to turn raw scheduler events into per-request profiles that expose off‑CPU waits—mutex contention, channel bottlenecks, scheduling latency, backoff sleeps and syscalls—without instrumentation. Demonstrates heuristics for lost edges and stack‑stitching, aggregates many requests into p50/p99 profiles, and reports production costs and gains (tracer overhead cut from ~10–20% to 1–2%). Shows a 110 ms request with 214 ms total mutex time and explains how the algorithm isolates what truly affects latency.
AI-written summary · about
chapters (8)
- 0:00Profiling request latency
- 1:30Profiling in production
- 4:00Limitations of profiling tools
- 8:12Execution tracer for context
- 13:09Critical path analysis
- 18:31Challenges in analysis
- 21:19Real-world results
- 23:25Future directions
related talks
38mFighting latency: the CPU profiler is not your ally
20mUnderstanding Request Latency with Wallclock Profiling
51mProduction Profiling: What, Why and How
44mGo Profiling and Observability from Scratch
35mTracing is More Than Traces: The Insights in Trace Aggregates
47mResolving performance issues in practice
26mProfiling Mastery: Decode Your Data and Elevate Observability
05mPerformance analysis using Systrace