Ray at Scale: Apple's Approach to Elastic GPU Management
summary
Apple runs Ray on Kubernetes with Apache YuniKorn queues and enforces per-queue guaranteed vs max GPU quota so teams get baseline GPUs while idle capacity is oversold and reclaimed via preemption. Gang scheduling is enabled with a CR label, bin‑packing uses YuniKorn scheduling plus a Carpenter post-scheduler to defragment nodes, and checkpointing plus reserved headroom reduce eviction failures. The pipeline is upstreamed to Kubernetes, runs in production, and materially raises utilization while preserving per-team guarantees.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 1:11Why use Ray
- 1:50How users use Ray
- 2:36Challenges using Ray
- 3:26GPU utilization
- 4:26Standard computer options
- 6:04Autoscaling
- 6:52Raise Autoscaler
- 7:36Challenges
- 10:26GPU Management
- 10:52Why Unicorn
- 12:24Unicorn Recap
- 13:09Unicorn Upstream
- 14:43Resource Queues
- 16:18Features
- 18:28Example Configuration
- 18:56Managing GPU Types
- 20:09Pin Packing
- 21:43Unicorn Schedule
- 24:28Demo
- 27:01Simulation
- 28:37Summary
- 30:18Outro
related talks
19mScaling Frontier AI with Ray on Kubernetes
28mRay on Public-Cloud k8s - experiments, lessons & best practices
32mRay in Production: Engineering Lessons from Spotify's Hendrix Framework
39mSailing Ray Workloads with KubeRay and Kueue in Kubernetes
27mEvolving and Scaling Ray for the Next Generation of AI Workloads
09mLightning Talk: Optimize the Ray Schedule and Autoscaling on Kubernetes Cloud: A Hetero...
26mIntroduction to Distributed ML Workloads with Ray on Kubernetes
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT