Hardware Latencies: How to Flush Them Out

on YouTube ↗50m

" summary

A 40‑core server shows sporadic 500 µs jitters that only appear after hours; investigation uses cyclictest (dash n, -p95, -b breakpoints), ftrace / function_graph via trace-cmd, and visualization with kernelshark. Tracing isolates a 235 µs spinlock stall from a real‑time push/pull thundering‑herd; fix replaces the lock-heavy notification with an IPI handoff. A modified hwlat_detector (threaded, outer/inner checks) then catches periodic SMIs (~5–10 µs) that when back‑to‑back sum to ~200 µs; both kernel and hardware causes get resolved.

AI-written summary · about

" chapters (16)

1. 0:00Start
2. 4:07Verification of Hardware
3. 6:12Latencies
4. 7:38The Enemy
5. 17:09Latency Tracers
6. 19:12Wakeup-rt
7. 21:17preemptirqsoff
8. 22:43Modified cyclictest
9. 23:45Latency without Load Balance?
10. 24:41Graph vs Function Tracing
11. 25:29What to do?
12. 28:08The Finding Nemo Seagull Effect!
13. 30:35HW Lat Detector
14. 35:00trace-cmd
15. 35:47kernelshark
16. 36:45Demo