High Perfomance Pandas
summary
Shows concrete Pandas patterns and timings: building a DataFrame from a dict with pd.DataFrame.from_dict(..., orient="index") cuts load from ~30s to ~3s for 10k rows; use append/concat for bulk adds not single-row appends; prefer vectorized apply/assign/json_normalize + pd.merge(how="inner") over building lists and iterating; use the add operator or fillna (≈7ms) instead of loc/iloc (≈148ms) for inplace edits; query outperforms loc after ~100k–200k rows. Notebook: https://github.com/LuisDavidCamacho/high-performance-pandas
AI-written summary · about
chapters (18)
- 0:00Introduction and setup
- 1:08Why pandas performance matters
- 2:55Efficient data frame creation
- 5:02Fast row iteration with apply
- 7:33Combining data sets cleanly
- 13:46Custom calculations without loops
- 19:50Complex data extraction from objects
- 20:13Fast data manipulation with operators
- 23:03Multi-index for faster queries
- 28:56Query method vs loc for speed
- 33:00Arithmetic and statistical operations
- 35:53Built-in data visualization
- 37:15Exporting data to LaTeX
- 37:37JSON normalization tips
- 38:21Loading data from URLs
- 39:02Splitting data by type
- 40:31Window and expanding methods
- 43:12Conclusion and resources
related talks
35mFast Pandas Performance Tricks You Wish You Knew Earlier
28mIdiomatic Pandas
32mPandas May Be Slow, but Pandas Doesn't Have to Be
35mIdiomatic Pandas
01h 02mPandas from the inside
31mFaster Pandas: Speed Up Your Code, Shrink Your Cloud Bill
01h 17mPandas from the Inside / "Big Pandas"
29mMaking Pandas Fly