Anyscale's Ray Data: Revolutionizing Batch Inference
summary
Ray Data plus Anyscale runtime implements a streaming executor with actor-based GPU workers and an optimizer, claiming Ray LM batch as a proprietary library that runs on spot instances and auto-tunes engine configs. Real results: 2,000 PDFs → ~40,000 embeddings on 20 GPUs in 4 minutes (~$0.95), 1.68 TB Falcon dataset on 320 GPUs in 18 hours (~$6,000), image batch runs up to 2× faster than Spark and 17× faster than SageMaker, and benchmarks report up to 2–6× cost improvements versus Bedrock with 100% GPU utilization.
AI-written summary · about
chapters (6)
- 0:00Introduction to talk
- 0:45What is batch inference
- 3:54Why Ray and Anyscale
- 11:21Code examples and results
- 14:50Introducing Ray LM Batch
- 21:38Summary and Q&A
related talks
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery
20mThe Quick Journey to Using Ray: How We Implement Ray and Anyscale to Speed up our ML Processes
30mScaling Machine Learning at Discord: Our Journey with Ray and Anyscale
14mFrom Reservation to 1,600 GPUs: A Petabyte-Scale Ray Data Pipeline
29mRay Data on GPU - Breaking Preprocessing Bottleneck for Model Training
21mRay Train on Anyscale: Observability and Reliability at Scale
26mData Processing on Ray
28mScaling Multimodal Data and Reinforcement Learning with Ray