Distributed XGBoost on Ray
summary
XGBoost-Ray implements a Ray-based backend that uses Ray actors and Rabit all-reduce to run multi-node, multi-GPU XGBoost with locality-aware loading from Parquet/HDFS/S3, seamless Ray Tune integration, and fault tolerance modes: warm restart and elastic training. A 30M-row, 500-feature benchmark over 100 rounds shows baseline (10 workers) ~13.3% eval error in ~20 minutes; elastic training keeps runtime near baseline with only a slight accuracy hit (30M rows 500 features, 13.3% eval error). Uber integrates it via a Spark estimator that materializes Parquet and spins Ray clusters.
AI-written summary · about
chapters (26)
- 0:00Intro
- 0:41Motivation
- 1:19XGBoost-Ray
- 1:53Recap: XGBoost
- 2:36Recap: Distributed XGBoost
- 3:22Architecture
- 4:20Distributed data loading
- 5:57Fault tolerance: Simple (cold) restart
- 6:24Fault tolerance: Non-elastic training (warm restart)
- 6:51Fault tolerance: Elastic training
- 7:24Fault tolerance: Benchmarks
- 9:03Hyperparameter tuning
- 11:43Distributed ML and DL Challenges at Scale
- 13:41Ray. A General Purpose Distributed Framework 1. Open Framework
- 16:22End-to-end ML at Uber
- 17:17Bridging the boundary between compute
- 18:05Spark ML Pipeline Abstractions
- 19:06Contract between Training and Serving
- 22:45How do we typically use the Ray Estimator?
- 23:00Prepare your Data with Spark
- 23:20Creating the Ray Context
- 23:48Create the Ray Estimator
- 24:10Save and Load Models
- 24:30Hyperparameter Optimization with Ray Tune
- 25:15Elastic Training + HyperSched
- 26:05Looking ahead: Unified Compute Infrastructure for ML and DL
related talks
29mAdopting Ray @ LinkedIn
30mScaling Machine Learning at Discord: Our Journey with Ray and Anyscale
32mHow Uber Optimize Marketplaces with Ray
32mAnyscale's Ray Data: Revolutionizing Batch Inference
26mUsing Ray On Large-scale Applications at Ant Group
01h 31mIntroduction to Ray for distributed and machine learning applications in Python
33mOvercoming Distributed ML Challenges with Ray Train
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT