How to Scale Unstructured Data Processing with Ray Data
summary
Ray Data on Anyscale claims to solve unstructured workloads by combining a streaming batch model with adaptive scheduling so GPU and CPU stages run concurrently, maximizing utilization and showing near-linear throughput as nodes scale. The platform demos a Stable Diffusion preprocessing pipeline with autoscaling to thousands of GPUs, checkpointing/resume and a failure‑aware scheduler, reports up to 90% efficiency in customer runs and an 4.5x faster job start from incremental metadata, with real‑time observability and resource-level metrics.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:38Use Cases for Unstructured Data
- 3:06Challenges with Unstructured Data
- 10:52Architecture of Ray Data
- 16:08Retable Data
- 20:44Example
- 25:05QA
related talks
26mData Processing on Ray
22mLarge Scale Data Loading and Data Preprocessing with Ray
20mThe Quick Journey to Using Ray: How We Implement Ray and Anyscale to Speed up our ML Processes
31mFrom OOMs to 2x Throughput: Scaling Multimodal Data Curation on Ray
29mRay Data on GPU - Breaking Preprocessing Bottleneck for Model Training
14mFrom Reservation to 1,600 GPUs: A Petabyte-Scale Ray Data Pipeline
35mOperating a Production-Scale Physical AI Data Platform with Ray
24mBuilding Complex Data Analytics Pipelines with Ray