Understanding Request Latency with Wallclock Profiling
summary
Shows that wallclock profiling (not just CPU sampling) reveals off‑CPU causes like GC allocation stalls, locks and blocking I/O and can change the optimization target for P99 latency. Implements a Datadog Java profiler that combines async-profiler, JFR and JVMTI, snapshots every 60s, updates trace context in under 10 ns, and uses a reservoir sample budget biased to traced request threads to avoid millions of samples. Uses a thread timeline view to lay out critical paths; heap + CPU resizing cuts P99 latency drastically.
AI-written summary · about
chapters (10)
- 0:00Introduction to profiling
- 0:56Identifying bottlenecks
- 2:15Limitations of flame graphs
- 3:18Wallclock profiling
- 5:33Thread timeline analysis
- 7:11Request-scoped diagnostics
- 10:02Resource impact on latency
- 11:58Profiling implementation
- 16:03Efficient sampling strategy
- 19:06Conclusion and wrap up
related talks
25mProfiling Request Latency with Critical Path Analysis
38mFighting latency: the CPU profiler is not your ally
44mJVM Profiling under the Hood
40mProfiling Without Pain: Always-On Performance for the JVM
51mProduction Profiling: What, Why and How
46mProfiling your Java Application A Beginner’s Guide
45mJava Performance Analysis and Optimization
48mFantastic Performance and Where to Find It