Extending Awkward Array into the broader PyData Ecosystem

on YouTube ↗29m

" summary

Provides native support for Awkward Array inside PyData by shipping dask-awkward and awkward-pandas. dask-awkward exposes a partitioned Awkward Array that scales computations to Dask schedulers, demonstrated on ~25 Parquet files (~<1 GB) to compute trip path lengths and a 2D histogram, with Dask graph optimizations collapsing many tasks. awkward-pandas supplies a Pandas extension array and accessor so nested columns run compiled Awkward ops inside DataFrame workflows, claiming massive acceleration and available as prereleases on PyPI.

AI-written summary · about

" chapters (9)

1. 0:00Extending Awkward Array
2. 1:49Awkward Array for nested data
3. 4:18Dask overview
4. 8:08Pandas nested data issues
5. 10:33Dask and Pandas integrations
6. 14:24Using new integrations
7. 15:28Demo: Dask + Awkward on taxi data
8. 24:58Demo: Pandas + Awkward on physics data
9. 28:54Closing & Q&A

" related talks

• 01h 22mUsing Dask for Parallel Computing in PythonSkipper Seabold · PyData Chicago 2016
• 01h 03mDask: Next Steps in Parallel PythonMatthew Rocklin · PyCon.DE 2017
• 44mUsing Dask for Parallel Computing in PythonMatthew Rocklin · PyData DC 2016
• 01h 26mDask tutorialPatrick Hoefler · PyData NYC 2023
• 31mThe Arrow revolution in pandas and DaskPatrick Hoefler · PyData NYC 2023
• 01h 17mScaling PyData with DaskMartin Durant, Jim Crist · PyData DC 2018
• 44mDask for ad hoc distributed computingMatthew Rocklin · PyData DC 2016
• 30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and PolarsPatrick Hoefler · PyCon US 2024