akimbo: vectorized processing of nested/ragged dataframe columns
summary
Library akimbo brings a numpy-like API to Arrow-backed dataframe columns, letting awkward-style nested/ragged arrays be processed without Python iteration. Demo shows a ~42k-package repo dataset where 385 packages require numpy 2.x, counted by applying vectorized tests instead of exploding rows. Numba-backed kernels yield about 40× speedups over pure Python loops; explode produces similar speed but much higher memory. Integrations include pandas, polars, dask-like backends and optional GPU/CPU execution, with slicing, ufuncs and aggregations at any nested level.
AI-written summary · about
chapters (11)
- 0:00Introduction to Akimbo
- 0:28Nested data in standard dataframes
- 1:54Arrow and nested formats
- 4:03Awkward Array and Akimbo
- 7:43Demo: Basic pandas usage
- 15:58Akimbo features explained
- 18:05Demo: HDFS logs with nested data
- 24:09Nested data sources and use cases
- 25:34Akimbo vs SQL or iteration
- 26:36Q&A: Arrow and pandas
- 28:48Closing remarks
related talks
31mThe Arrow revolution in pandas and Dask
24mPolyglot data with python: Introducing Pandas and Apache Arrow
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
29mExtending Awkward Array into the broader PyData Ecosystem
30m"Better dataframes"
09mBackstage interview
46mInternals of Speeding up PySpark with Arrow
39mDesigning for high-performance execution with Arrow and Polars