Structuring Data for Scalable AI Applications
summary
Open-source Data Chain turns terabytes of unstructured files into a small SQL table of pointers and metadata, versioned and queried via a Python API that maps operations to SQL, enabling filtering, joins, train/validation splits and conversion to PyTorch Datasets without copying bytes. It runs map/gen jobs that stream from S3, generates embeddings on GPU clusters and supports multimodal rows (image/video/text/PDF). R&D shows parallel processing at millions-file scale and examples where dataset size is halved with near-identical accuracy.
AI-written summary · about
chapters (14)
- 0:00Unstructured data management
- 1:31AI adoption challenges
- 3:20Unstructured data formats
- 4:24Challenges in data pipelines
- 7:36Data Chain tool for AI
- 8:42How Data Chain works
- 11:58PDF processing with LLMs
- 15:13Computer vision for fashion
- 17:26Training models with Data Chain
- 19:33Embeddings and GPU acceleration
- 21:00Reducing dataset size
- 22:03Video analytics and frames
- 24:11Data Chain benefits
- 25:15Q&A
related talks
14mSoftware 2.0 Needs Data 2.0: A New Way of Storing and Managing Data for Efficient Deep Learning
31mBuilding a Data Infrastructure for AI/ML
58mAI-Enhanced Big Data: Integrating Private LLMs and Vector Databases
18mData Preprocessing for ML at Production Scale
26mOne Pipeline to Rule Them All: Unifying Multimodal and AI Data Processing with Daft
45mManaging Data & Analytics Projects Effectively
23mDaft - The Distributed Python Dataframe for Complex Data
37mTaming the Data Tsunami