Beware of the kernel RTNL mutex
summary
The global RTNL mutex serializes all kernel networking control-plane changes and can turn routine queries into multi-second stalls — reproducible case shows steady 100–200 ms latency spiking to ~3 s when deleting a pod triggers many syncs (one driver did ~3,000 ops). Use klockstat (libbpf-tools) with stack traces and netlink/ioctl decoding to see who holds/waits on the lock; mitigation is to move queries off the fast path (background thread or cache) or convert readers to RCU conversion and rely on upstream per-namespace / per-device locking work.
AI-written summary · about
chapters (18)
- 0:00Kernel RTNL mutex intro
- 1:25Customer escalation reveals issue
- 3:29What is RTNL mutex
- 5:17Debug RTNL with klockstat
- 9:09Analyze lock holders
- 10:52Enhanced klockstat stack traces
- 12:37Mitigate RTNL blocking
- 14:23Key takeaways design
- 15:49Fix customer issue with RCU
- 16:31Reduce RTNL scope efforts
- 21:26Q&A: Documentation blocking
- 23:15Q&A: klockstat debugging
- 24:16Q&A: Read-write semaphore
- 25:20Q&A: Extreme driver blocking
- 26:35Q&A: High-rate queries
- 27:41Q&A: Statistics RCU efforts
- 28:46Q&A: RTNL upstream steps
- 31:14Q&A closing
related talks
24mPREEMPT_RT Q&A with tglx
51mTuning RT Kernel to Improve Schedule Performance with Intel Platform
52mEffectively Measure and Reduce Kernel Latencies for Real-time Constraints
50mLow-Latency Real-Time (RT) Kernel: The Cornerstone of Arm's 5G ORAN Strategy
46mPREEMPT_RT - how not to break it.
26mReal-time Linux: what is next?
50mrtla timerlat: Debugging Real-Time Linux Scheduling Latency
53mPREEMPT_RT - How Not to Break It.