Graphing tools for scheduler tracing
summary
Trace-cmd traces scheduler events and the new graphing tools summarize them: dat-to-graph draws per-core run intervals across 128 cores so gaps show unused CPUs while others overload, and running-waiting plots running vs waiting threads revealing a persistent one-thread shortfall that stretches tens of seconds. Examples use a 4-socket 128-hardware-thread NAS benchmark where Linux 6.5 worst runs ~40s, 6.6 with ESVDF spikes to ~85s, NUMA balancing counts ~600 vs ~459 and time slices collapse to one tick under ESVDF.
AI-written summary · about
chapters (13)
- 0:00Introduction to scheduler tracing
- 1:22Goals and tools overview
- 2:25Benchmark setup and pinning results
- 4:12Performance regression in Linux 6.6
- 5:17Introducing dat to graph tool
- 6:20Comparing 6.5 and 6.6 with dat to graph
- 7:46Introducing running waiting tool
- 8:49NUMA balancing analysis
- 10:34NUMA balancing in Linux 6.6
- 11:37Time slice changes with evdf
- 13:43Testing time slice modifications
- 14:50Unknown performance issue
- 15:32Q&A and future directions
related talks
35mOpening the Box: Diagnosing Operating-System Task-Scheduler...
18mScheduler Tracing With ftrace + eBPF
38mThe Structure and Interpretation of Graphs
55mAdvanced Tracing Features to See What Your Application is Waiting On
44mNow You See Me Too: Visual Tooling for Advanced System Analysis
46mOptimizing Scheduler for Linux Gaming
51mLinux Systems Performance - Architecture
57mParallel Computing Visualization