Debugging Methodologies for Realtime Issues in Linux Systems
summary
Shows how to find and fix real latency sources using concrete tools and fixes: uses irqsoff tracer and function_graph tracer to spot top-half handlers and preempt-off windows, inserts kprobes to time irq handlers, and uses cyclictest and latency_hist to validate results. Finds a 14 ms wakeup/IPI delay, a bus PCIe low-power bug fixed by disabling PCI power management, reduces a VM lazy-free threshold from 32MB to 8MB to avoid 14 ms stalls, and converts long nonthreaded IRQs to threaded IRQs which eliminates the observed pauses.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 6:47Preemption
- 8:05RT Patch
- 11:25Kernel Off Tracer
- 19:14Recommendations
- 21:03Preempt Arceus Off Tracer
- 24:07Hardware Issues
- 36:39Scheduling Issues
- 39:19Cyclic Tests
- 42:50Latency Tracker
related talks
52mEffectively Measure and Reduce Kernel Latencies for Real-time Constraints
26mTuning Real-Time Linux for ARM SoCs
44mGetting Around to It: Deferred Work in the Linux Kernel
32mCPU Power Saving Methods for Real-time Workloads
50mrtla timerlat: Debugging Real-Time Linux Scheduling Latency- 50mrtla timerlat: Debugging Real-Time Linux Scheduling Latency
26mReal-time Linux: what is next?
36mJourney of improving RT performance on Single CPU equipped PREEMPT_RT Linux