When Linux Memory Accounting Goes Wrong
summary
LinkedIn searcher nodes hit host OOMs because cgroup page-cache accounting fails when a container is restarted and its cgroup is deleted and recreated: shared pages remain in RAM but the new cgroup doesn’t claim them. Observed numbers: host shows ~51 GB cgroup usage while application metrics sum to ~63 GB (page cache 44 GB vs cgroup 31 GB). The team reproduces behaviors with a C tester, notes mlockall prevents swapping, adds application index size + RSS monitoring and sets a limit on the parent cgroup as a mitigation.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:32Search at LinkedIn
- 1:52Concepts
- 4:18The Problem
- 7:06Initial Debugging and Resolution (conto)
- 8:33Issue recurrence
- 11:43Troubleshooting (contd)
- 15:24Memory accounting in cgroups
- 17:58Validation
- 19:06What happened in our Case
- 22:00Mitigation
- 24:33Conclusion
related talks
01h 36mTutorial: Linux Memory Management and Containers- 01h 36mLinux Memory Management and Containers
26mLet's demystify memory
41mLies Programmers Believe about Memory
38mSolving Zombie Cgroups - Simple Fix or Smart Solution?
32mWho Killed My Pod? #Whodunit
20mChasing Bad Checksums: A Journey Through Ceph, TCMalloc, and the Linux Kernel
29mTrip Down Memory Lane: OOM in Python