Faster Pandas: Speed Up Your Code, Shrink Your Cloud Bill
summary
Shows concrete wins: switching CSV to parquet with pyarrow drops an 8.6s CSV load to ~283ms (≈28x); using list appends plus one concat or pandas.from_records with a generator cuts a 712ms loop to ~12ms (≈56x); replacing pandas max with a NumPy-based max yields ~100x speedups in that case. Demonstrates profiling toolchain: %timeit, timeit, cProfile / prun, pytest-benchmark and visualizers like snakeviz, and highlights memory counts (a DataFrame at ~601MB) and avoiding CSV to save RAM and cloud cost.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker intro
- 1:39Why optimize Pandas performance
- 2:22Rules of the optimization club
- 8:08Loading data efficiently
- 12:56Benchmarking with timeit
- 15:54Profiling to find bottlenecks
- 19:37Visualizing profiling results
- 21:04Optimizing DataFrame concatenation
- 23:15Using numpy for faster calculations
- 25:10Memory optimization strategies
- 29:07Key takeaways and Q&A
related talks
40mBeyond Pandas: The Great Python Dataframe Showdown
35mFast Pandas Performance Tricks You Wish You Knew Earlier
32mPandas May Be Slow, but Pandas Doesn't Have to Be
29mMaking Pandas Fly
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
44mHigh Perfomance Pandas
01h 17mPandas from the Inside / "Big Pandas"
33mSkinny Pandas Riding On A Rocket