Scaling Massive Transformer Training with KubeRay
summary
Capital One runs KubeRay on Kubernetes to stream 3.5TB (≈2 billion records) from S3 into GPUs and consolidates Ray Data, Ray Train and Ray Tune into one platform. Result: 16× hyperparameter tuning speedup, 20× data pipeline acceleration versus legacy loaders, 1.6× training throughput over vanilla PyTorch DDP and 3× tuning velocity with 200 concurrent trials plus early pruning. Built features: lazy streaming from the Ray object store to GPU, zero-copy sharding to workers, pod autoscaling and pre-GPU memory limits to prevent OOMs.
AI-written summary · about
chapters (9)
- 0:00Introduction and vision
- 1:10AIML platform challenges
- 4:03Why we chose Ray
- 7:58Transformer training bottlenecks
- 10:29Ray Data for streaming
- 11:34Ray Train distributed scaling
- 12:41Ray Tune hyperparameter search
- 14:06Unified Ray ecosystem results
- 14:49Key lessons and best practices
related talks
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT
35mHow Bytedance Builds Large-Scale Data Processing Pipelines for Multimodal Models with Ray
35mReddit's ML Evolution: Scaling with Ray and KubeRay
35mScaling Ray Train to 10K Kubernetes Nodes on GKE
22mLarge Scale Data Loading and Data Preprocessing with Ray
27mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case St...
28mScaling Multimodal Data and Reinforcement Learning with Ray
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery