RayDP: Build Large-scale End-to-end Data Analytics and AI Pipelines Using Spark and Ray
summary
RayDP runs Spark on Ray by launching Spark driver and executors as Ray Java actors, uses the Ray in-memory object store to exchange DataFrames without HDFS, and provides a converter to make Spark DataFrames into a Ray ML dataset for downstream libraries. It exposes a PyTorch/TensorFlow estimator for one-call distributed training, integrates XGBoost and Horovod, supports Ray Tune for hyperparameter search, and scales from laptop to cloud with Ray cluster auto-scaling.
AI-written summary · about
chapters (9)
- 0:00Introduction to RayDP
- 1:26Big Data and AI Integration
- 3:55Challenges in Traditional Setups
- 7:11Ray and RayDP Solution
- 10:20Building Pipelines with RayDP
- 14:34RayDP API and Architecture
- 17:21Estimator API for Training
- 20:21RayDP Examples
- 24:55Summary and Resources
related talks
03mRay: A Distributed Execution Framework for Emerging AI Applications
31mNext-Generation Data Science Workflows Using Ray
22mRay, a Unified Distributed Framework for the Modern AI Stack
24mBuilding Complex Data Analytics Pipelines with Ray
33mOvercoming Distributed ML Challenges with Ray Train
35mOperating a Production-Scale Physical AI Data Platform with Ray
24mRay: A System for High-performance, Distributed ML Applications
01h 31mIntroduction to Ray for distributed and machine learning applications in Python