Python Data Ecosystem: Thoughts on Building for the Future
summary
Discussion centers on concrete work not platitudes: a C++11 library and bindings that implement a columnar in‑memory format called Arrow, a tiny file format Feather that memcopies a 1GB data frame between R and Python in seconds, an Apache Parquet adapter in C++ under development, and automation for binary packages via conda-forge. Also covered are hard facts about packaging, multi-process memory sharing, conversion costs, and community mechanics like consensus and public mailing‑list decision making.
AI-written summary · about
chapters (7)
- 0:00Python data ecosystem future
- 4:18Building open source communities
- 17:24Community challenges and solutions
- 24:45Python packaging challenges
- 32:23Collaborating with Arrow and Feather
- 49:40Arrow and Feather interoperability demo
- 53:12Q&A
related talks
24mPolyglot data with python: Introducing Pandas and Apache Arrow
14mKeynote: Scalable Python We Can Afford
41mKeynote: Making the Big Data ecosystem work together with Python
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
40mBeyond Pandas: The Great Python Dataframe Showdown
31mThe Arrow revolution in pandas and Dask
08mBackstage interview
47mPanel: Fostering better collaboration between the scientific Python community and core Python...