There and Back Again: Disable and re-enable garbage collector at Instagram
summary
Instagram measures the problem with perf profiling, finds garbage collection causes page faults that trigger copy‑on‑write and costs about 150 MB per process saved when GC is disabled, with CPU throughput up ~10% and memory utilization down ~15%. Disabling GC leaks ~600 MB after ~3,000 requests, so Instagram implements a GC.freeze API and a workflow that full‑collects and freezes pre‑fork objects then enables GC in children; the change lands in upstream Python (3.6/3.7) and works.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:30ACENDA
- 1:07MEMORY MANACEMENT IN PYTHON
- 2:53COPY-ON-WRITE SEMANTIC
- 3:42INSTACRAM TECH STACK
- 4:37OBSERVATION
- 5:38PROFILINC
- 6:16WHY GARBACE COLLECTION?
- 8:31DISABLE CC
- 12:23AFTER ONE YEAR
- 13:44REDESIGN CC HEAD
- 15:28POINTERS TAHE SPACE
- 16:15FREEZE OBJECTS
- 18:18MEMORY POOL
- 19:19RESULTS
- 20:16WRAP UP
related talks
23mMemoirs of a Python Object: Memory Management and Improving Performance
26mPython Performance at Scale - Making Python Faster at Instagram
27mDemystifying Python’s Generational Garbage Collector
25mIs it me or Python memory management? 🤔💻
29mTrip Down Memory Lane: OOM in Python
31mIntroducing Immortal Objects: Building Block Towards a Multi-core Python Runtime
29mPython's Infamous GIL
40mKnowing your garbage collector