Latency Distributions and Micro-Benchmarking to Identify and Characterize Kernel Hotspots

on YouTube ↗26m

" summary

Engineering uses micro-benchmarks and latency histograms to pinpoint kernel hotspots across AIX Linux and Solaris rather than rely on topic-level diagnostics. AIX reveals semaphore create/destroy serialization via coarse spinlocks that lengthen wall time under concurrency confirmed with a 3 million iterations create/destroy loop and aggravated by AIX hyperthreading CPU accounting. Linux message queue shm shows huge system time that improves from RHEL6 to RHEL7.6. Solaris netsat traversal of unix domain sockets pushes a token SLI tail from median ms to 250 milliseconds; a benchmark that adds 32k sockets per run reproduces the growth and leads to Solaris 11.4 fixes.

AI-written summary · about

" chapters (17)

1. 0:00Intro
2. 2:11Why Large Bare Metal Boxes? • Faster local communication UNIX Domain Sockets Shared Memory
3. 2:58The Scale in our Department • 100K processes across hundreds of physical machines
4. 3:22SysV semaphore bottleneck (AIX)
5. 8:07Observations and Findings AIX CPU measurement when hyper-threading is very misleading No 'out of the box metrics on SysV IPC operations Sporadic slowness (depending on concurrency/contention)
6. 9:04SysV shared memory bottleneck (Linux) • Low-level application infrastructure code dropping messages Messaging leverages a form of "zero copy" IPC using Sysv
7. 9:39SysV shared memory bottleneck (Linux RHEL 6) The micro-benchmark
8. 10:37Case #2: Observations and Findings • No 'out of the box metrics on SysV IPC operations
9. 11:59UNIX domain socket bottleneck (Solaris) • Critical software infrastructure experiencing timeouts on load Identity management with very strict SLOS Narrowing down the problem A key SLI for the service is token generation latency
10. 12:51An Aside: Histograms and Distributions are Useful! • More representative of the data set
11. 14:18An Aside: A Histogram Example
12. 14:43Early Observations • No out of the box metrics on socket operations
13. 16:40Case #3: UNIX domain socket bottleneck (Solaris) The micro-benchmarkt-testing against size
14. 17:41Case #3: Conclusions • Solaris 11.3 is limited to a max of 256K UDS sockets
15. 18:23Task clone and exit bottleneck (Linux)
16. 22:37More Summary (Plea to Kernel Folks) • The Prime Directive of Monitoring: Non-interference
17. 24:34References

" related talks

• 41mLinux Systems PerformanceBrendan Gregg · LISA19
• 35mOpening the Box: Diagnosing Operating-System Task-Scheduler...Julia Lawall · SREcon24 Europe/Middle East/Africa
• 51mLinux Systems Performance - ArchitectureBrendan Gregg · Percona Live 2016
• 49mDebugging Methodologies for Realtime Issues in Linux SystemsJoel Fernandes · Embedded Linux Conference + OpenIoT Summit Europe 2016
• 52mEffectively Measure and Reduce Kernel Latencies for Real-time ConstraintsChung-Fan Yang · Embedded Linux Conference + OpenIoT Summit North America 2017
• 25mShenango: Achieving High CPU Efficiency for Latency-sensitive Datacenter WorkloadsJoshua Fried, Adam Belay, Amy Ousterhout, Hari Balakrishnan, Jonathan Behrens · NSDI 2019
• 21mGraphing tools for scheduler tracingJulia Lawall · Linux Plumbers Conference 2023
• 50mrtla timerlat: Debugging Real-Time Linux Scheduling LatencyDaniel Bristot de Oliveira · Real-Time Linux Summit 2024