Processing large JSON files without running out of memory

on YouTube ↗37m

" summary

Loading a 50 MB JSON file can use ~230 MB of RAM (about 4.5×) and a 10 GB file can demand ~40 GB, so peak memory is the limiter not average. Four practical fixes appear: buy more RAM, use compact in‑memory formats (awkward/Arrow → polars/pandas), only decode needed fields (message spec / pydantic tradeoffs), or process by streaming/chunking (iJSON, NDJSON lines) to keep memory bounded.

AI-written summary · about

" chapters (12)

1. 0:00Introduction
2. 1:10Ethical considerations
3. 3:58Memory usage matters
4. 6:43Measuring memory in Python
5. 7:26Four solutions
6. 10:39Efficient in-memory representations
7. 14:51Load only needed data
8. 17:39Streaming JSON files
9. 22:18Alternatives to JSON
10. 25:08Why measure memory
11. 26:09Recap of strategies
12. 26:54Q&A

" related talks

• 30mProcessing large JSON files without running out of memoryItamar Turner Trauring · PyCon US 2025
• 01mSmall Big Data: using NumPy and PandasItamar Turner Trauring · Data Love 2021
• 26mSmall Big Data: using NumPy and Pandas when your data doesn't fit ...Itamar Turner Trauring · PyCon US 2020
• 40m640K ought to be enough for anyoneYonatan Levin · droidcon Berlin 2022
• 33mSkinny Pandas Riding On A RocketIan Ozsvald · PyData Global 2020
• 32mOptimize Your JSON Payload Efficiency x10 timesGireesh Punathil · Node+JS Interactive 2018
• 50mWe Must Save the Last Giga of RAMIvan Béthus, Damien Lucas · Voxxed Days Luxembourg 2024
• 35mYour data fits in RAM: How to avoid cluster computingAaron Richter · PyData Miami 2019