Keynote: Scalable Python We Can Afford

on YouTube ↗14m

" summary

Pandas often requires 10× RAM because data must be RAM resident and many algorithms remain single threaded and inefficient for strings; scaling by naively distributing pandas multiplies serialization and network overhead. A practical fix is Apache Arrow, a language‑independent columnar format and in‑memory API used by Dask, Ray and NVIDIA Rapids; Rapids benchmarks show nearly 20× faster execution with one‑third the power, roughly a 60× compute efficiency improvement versus prior approaches.

AI-written summary · about

" chapters (10)

1. 0:00The origins of pandas
2. 0:39Passion for open source
3. 1:18Building the toolkit
4. 1:50Community and design vision
5. 2:31Technical debt and scale
6. 2:49Addressing performance issues
7. 3:50Challenges in big data
8. 7:20The Apache Arrow project
9. 12:32Efficiency and performance
10. 12:55The future of data tools

" related talks

• 43mKEYNOTE: Scaling Out PyDataTravis E. Oliphant · PyData London 2016
• 41mKeynote: Making the Big Data ecosystem work together with PythonHolden Karau · PyData London 2018
• 40mScaling Python - Bank EditionAndrew Fulton, Adam Lewis · PyData NYC 2022
• 40mBeyond Pandas: The Great Python Dataframe ShowdownJuan Luis Cano Rodríguez · PyData London 2022
• 31mThe Arrow revolution in pandas and DaskPatrick Hoefler · PyData NYC 2023
• 24mPolyglot data with python: Introducing Pandas and Apache ArrowRobson Luis Monteiro Junior · PyCon US 2020
• 58mPython Data Ecosystem: Thoughts on Building for the FutureWes McKinney · PyData Berlin 2016
• 34mSmall Big Data: using NumPy and Pandas when your data...Itamar Turner Trauring · PyData New York City 2019