"Logs Told Us It Was Kernel – It Wasn't"
summary
Investigation shows a 30% slowdown on a tiny loop and a clock_gettime jump from 17ns to 47ns that falls back to 23ns after boot-parameter fixes; root causes include hundreds of cgroups, a changed memcpy implementation in gpsy/JPC vs GPC, and poor instruction-cache alignment created by GCC 8.5 code layout. Tools used: Intel VTune, disassembly, register hints, adding nops to align loops, and profile‑guided compilation; practical fixes are reducing cgroups, switching memcpy tunables, and using compiler flags like -falign-functions or PGO.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:18Case studies of performance
- 4:15Methodology and CPU concepts
- 6:20Memory architecture analysis
- 10:13Profiling and code optimization
- 15:34Cache alignment investigation
- 21:03Compiler optimization flags
- 25:21Profile guided optimization
- 28:08Conclusion and best practices
related talks
41mLinux Systems Performance
49mDebugging Methodologies for Realtime Issues in Linux Systems
35mOpening the Box: Diagnosing Operating-System Task-Scheduler...
52mEffectively Measure and Reduce Kernel Latencies for Real-time Constraints
32mIs Your Kernel Being Honest? Understanding & measuring low level bottlenecks
28mExtreme OS Kernel Testing
52mLinux Kernel Debugging: Going Beyond Printk Messages
39mCPU isolation vs jailbreaking IPIs