Daft - The Distributed Python Dataframe for Complex Data
summary
Daft claims a Python-native distributed dataframe that handles images video and other non-tabular types, using Apache Arrow Parquet and Iceberg for serialization and a one-line switch to a Ray cluster for execution. Demo shows loading 10,000 rows, filtering to 2,000 images, converting bytes to PIL, resizing to 256, running UDFs (including stateful GPU-capable ones), optimizer fuses filters and out‑of‑core execution spills to disk so the pipeline actually scales and finishes.
AI-written summary · about
chapters (12)
- 0:00Introduction to Daft
- 1:10What is Daft
- 3:40Complex data challenges
- 7:31Tools and limitations
- 8:58Daft’s solution
- 10:01Demo: loading image data
- 13:55Transforming strings to images
- 15:42Applying user-defined functions to images
- 16:45Detecting red pixels
- 17:46Scaling to distributed
- 20:55Summary of Daft
- 21:41Q&A
related talks
30mBlazing fast I/O of data in the cloud with Daft Dataframes
26mOne Pipeline to Rule Them All: Unifying Multimodal and AI Data Processing with Daft
32mConnecting PyData to other Big Data Landscapes using Arrow and Parquet
24mPolyglot data with python: Introducing Pandas and Apache Arrow
30mIs the great dataframe showdown finally over? Enter: polars
29mEfficient and portable DataFrame storage with Apache Parquet
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet
39mAccelerating Data Science with RAPIDS