Processing large JSON files without running out of memory

on YouTube ↗30m

" summary

Shows concrete numbers and tools: json.load -> 50 MB file becomes ~200 MB (4x) measured with user/bin/time, resource and tracemalloc. Demonstrates three fixes with real outcomes: awkward/Arrow -> peak ~100 MB (2x) but different API, message-spec schema -> ~100 MB but may still read whole file into memory, and ijson streaming -> ~1 MB peak independent of file size. Notes pandas/polars can use more memory, recommends measuring peak and choosing streaming, schema subsetting, or Arrow based on data shape and API needs.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 3:07JSON memory usage
3. 7:41Buying more RAM
4. 10:30Efficient in-memory representation
5. 15:49Load only needed data
6. 19:43Stream JSON efficiently
7. 25:38JSON alternatives
8. 29:31Summary

" related talks

• 37mProcessing large JSON files without running out of memoryItamar Turner Trauring · PyData Boston 2025
• 24mSerialization More than picklingJoseph Lucas · PyCon US 2022
• 33mSkinny Pandas Riding On A RocketIan Ozsvald · PyData Global 2020
• 40mBeyond Pandas: The Great Python Dataframe ShowdownJuan Luis Cano Rodríguez · PyData London 2022
• 01h 04mAPI Design as a Performance PrimitiveDavid Cheney · GopherCon UK 2026
• 01mSmall Big Data: using NumPy and PandasItamar Turner Trauring · Data Love 2021
• 36mThe Perils of Large FilesClaudio Ortolina · ElixirConf EU Virtual 2020
• 54mUnderstanding of Jsonb PerformanceOleg Bartunov · PGConf NYC 2021