Data Processing on Ray
summary
Ray adds concrete features to run large-scale ETL and ETL→ML pipelines: it implements memory-aware scheduling and locality-aware placement to avoid needless copies, a decentralized memory manager with admission control to prevent deadlocks, and object spilling to disk/S3 for out-of-core workloads. Integrations like Dask on Ray, Modin and XGBoost-on-Ray run in one cluster, showing a reported 100 TB distributed shuffle success and earlier 4× cost savings on a 4 PB job.
AI-written summary · about
chapters (9)
- 0:00Introduction to Ray
- 3:54Why Ray for data processing
- 6:18Ray’s data processing limitations
- 9:49Real-world data processing demo
- 11:54Key features for data processing
- 16:50Scheduling improvements
- 18:58Out-of-core data processing
- 21:43Memory management upgrades
- 24:50Results and future plans
related talks
24mBuilding Complex Data Analytics Pipelines with Ray
30mHow to Scale Unstructured Data Processing with Ray Data
26mUsing Ray On Large-scale Applications at Ant Group
31mFrom OOMs to 2x Throughput: Scaling Multimodal Data Curation on Ray
35mHow Bytedance Builds Large-Scale Data Processing Pipelines for Multimodal Models with Ray
15mRay: A Distributed Compute Engine for AI
29mDask-on-Ray: Using Dask and Ray to Analyze Petabytes of Remote Sensing Data
35mOperating a Production-Scale Physical AI Data Platform with Ray