One Pipeline to Rule Them All: Unifying Multimodal and AI Data Processing with Daft
summary
Daft is a Python-native, Rust-executed data engine that runs multimodal ETL, document parsing with Dockling, embedding via Hugging Face or OpenAI, and object-storage IO (S3/GCS) in one pipeline. A demo runs on Ray/Kubernetes with 768 CPUs and 64 GPUs and processed 1,000 PDFs in 2 minutes by streaming downloads, chunking, and batched embedding; runtime auto-tunes partitions and batching to avoid OOMs and claims 10x throughput vs Spark. Native and distributed runners work; the live job completes successfully.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 1:06Big Data to AI evolution
- 4:15Multimodal AI challenges
- 5:40Need for unified engine
- 6:21Document pipeline limits
- 8:27Scaling with object storage
- 9:33Introducing DAFF
- 10:14DAFF’s lazy execution
- 11:19Transition to distributed
- 11:40Document parsing with Dockling
- 13:04Embeddings and vector DBs
- 14:07DAFF execution modes
- 15:58Live demo: 1,000 PDFs on Ray
- 16:19DAFF vs Spark performance
- 18:25Streaming and memory optimizations
- 19:07Performance results
- 20:13Q&A: Rate limits
- 21:17Q&A: Cloud execution
- 22:22Q&A: Kubernetes autoscaling
- 23:24Q&A: Custom downloads
- 24:10Q&A: Fast S3 downloader
- 25:15Q&A: Throttling async
related talks
23mDaft - The Distributed Python Dataframe for Complex Data
19mAccelerating ML Development with Multi-Modal Datasets: Python, Parquets, and Daft
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case S...
22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case Study
46mScaling ML Pipelines with Feast, Ray and Kubeflow
29mDagster: a Framework for Data Processing Applications
38mData Prep Kit: A Comprehensive Cloud-Native Toolkit for Scalable Da...
39mAccelerating Data Science with RAPIDS