DataFrames: scaling up and out
summary
Pandas works in memory so the checklist is: avoid per-row Python loops and use vectorized ops to get 100x speedups; load only needed columns, use chunking or sampling, and cast strings to categoricals to reduce memory. Options scale from buying bigger machines (workstation with terabyte of RAM), to VPSes (DigitalOcean / AWS Lightsail ~ $20/mo) with SSH tunneling and S3 for storage, to query‑backends like Athena/Presto ($5 per TB processed), or horizontally with Dask/Spark (pip install pyspark) for true distributed compute.
AI-written summary · about
chapters (13)
- 0:00Scaling DataFrames
- 1:05Why developers love DataFrames
- 3:55Scaling challenges
- 5:19Optimizing pandas
- 8:08When to avoid pandas
- 10:40Streaming alternatives
- 11:22Scaling up machines
- 12:03Remote scaling with VPS
- 18:29Scaling the data source
- 21:17Cloud analytics engines
- 23:25Spark and Dask
- 27:40Trade-offs in distributed solutions
- 30:07Summary
related talks
26mSmall Big Data: using NumPy and Pandas when your data doesn't fit ...
31mStrategies for Working with Data as It Grows
53mEnterprise-Ready Python for High-Performance Data Teams
35mData Processing Pipelines for Small Big Data
36mScaling data workloads using the best of both worlds: pandas and Spark
30mScaling Outside the Warehouse Using DuckDB and Python
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
30mSpeed is Not All You Need for Data Processing