Pandas from the Inside

on YouTube ↗01h 21m

" summary

Slides and code dig into pandas internals using numpy arrays and real AFL game data and build a season ladder that matches the AFL website. Concrete claims include microbenchmarks: Python sum ~10 ms vs numpy sum ~1.2 ms and calling Python sum on a numpy array ~15× slower; column access: .values ~35 ms vs two-column frame ~0.5 ms vs single Series ~2 µs. Key mechanisms are block manager, take indexing, and cached series plus groupby/concat tradeoffs.

AI-written summary · about

" chapters (13)

1. 0:00Introduction
2. 4:59Python lists vs NumPy arrays
3. 15:42NumPy indexing deep dive
4. 22:15Pandas structure overview
5. 26:14Pandas internals overview
6. 30:35Loading AFL football data
7. 37:53Rebuilding AFL Premiership ladder
8. 54:19Reformatting columns and indexes
9. 1:02:21Multi-index operations
10. 1:07:05Efficient row selection
11. 1:11:03Calculated columns and warnings
12. 1:16:22GroupBy aggregation tips
13. 1:18:30Final results and key takeaways

" related talks

• 01h 17mPandas from the Inside / "Big Pandas"Stephen Simmons · PyData Berlin 2017
• 01h 02mPandas from the insideStephen Simmons · PyData London 2016
• 32mPandas May Be Slow, but Pandas Doesn't Have to BeAndrew Vaccaro · PyColorado 2019
• 29mMaking Pandas FlyIan Ozsvald · PyData Fest Amsterdam 2020
• 42mDemystifying pandas internalsMarc Garcia · PyData London 2018
• 40mBeyond Pandas: The Great Python Dataframe ShowdownJuan Luis Cano Rodríguez · PyData London 2022
• 33mThe Secret Life Of Rolling PandasJaime Fernandez del Rio · PyData Barcelona 2017
• 28mIdiomatic PandasMatt Harrison · Conf42 Python 2021