Tuning Memory and the Network Stack in Linux

on YouTube ↗36m

" summary

Migration of high-throughput video delivery servers (each pushing ~40 Gbit/s) from OpenSuSE 10.2 to CentOS 7 exposed issues: unexplained swapping and OOM killer runs, uneven CPU core load, connection drops, and CPU load spikes. The talk explains how those problems were diagnosed and mitigated by tuning Linux memory/subsystem behavior and the network stack, describing the specific tools and kernel/settings changes used.

AI-written summary · about

" chapters (13)

1. 0:00Introduction
2. 1:51Video server issues
3. 2:15Memory fragmentation fix
4. 6:57Swap and NUMA problem
5. 11:17Network tuning and broken pipe
6. 14:34Network load balancing
7. 18:48RSS, RPS, XPS, RFS
8. 20:57Accelerated RFS and Intel
9. 22:45Broken pipe consequences
10. 23:07Softirq CPU load
11. 27:42Gradual changes
12. 29:11Summary
13. 29:37Q&A

" related talks

• 55mPerformance tuning of Red Hat Enterprise LinuxLarry Woodman · DevConf.CZ 2019
• 51mLinux Systems Performance - ArchitectureBrendan Gregg · Percona Live 2016
• 52mPerformance tuninng for Red Hat Enterprise Linux.Larry Woodman · DevConf.CZ 2020
• 01h 36mTutorial: Linux Memory Management and ContainersGerlof Langeveld · Open Source Summit Europe 2023
• 23mBuilding a Network Operating System Using Linux and YoctoJohn Mehaffey · Open Source Summit Europe + Embedded Linux Conference Europe 2019
• 27mDebugging Linux Issues with eBPFIvan Babrou · LISA18
• 01h 36mLinux Memory Management and ContainersGerlof Langeveld · ContainerCon @ OSS EU 2023
• 25mShenango: Achieving High CPU Efficiency for Latency-sensitive Datacenter WorkloadsJoshua Fried, Adam Belay, Amy Ousterhout, Hari Balakrishnan, Jonathan Behrens · NSDI 2019