Maximize GPU Utilization for Model Training
summary
Uses PyTorch + Alluxio + S3 to build a distributed caching layer colocated with Kubernetes training clusters, implements consistent-hash affinity caching and worker-side metadata to remove a master bottleneck, and measures concrete wins: data loader time drops from ~82% to ~1% and GPU utilization rises from <20% to >90% on ResNet/ImageNet-style runs. Benchmarks with Ray show up to 5x throughput under memory pressure, and model rollout time improves by 2–10x depending on optimizations.
AI-written summary · about
chapters (17)
- 0:00Alasio data platform intro
- 1:30Data locality challenges
- 3:37Local caching limits
- 4:38Distributed caching master-worker
- 6:24Consistent hashing for caching
- 7:29Affinity caching load balance
- 8:51Multi-cloud training challenges
- 10:58Storage pressure from training
- 12:02Caching for train/serve clusters
- 13:30PyTorch benchmarking with Alasio
- 17:29Ray benchmarking with Alasio
- 19:37Faster model deployment
- 20:45Alasio compute frameworks
- 21:27Alasio AI evolution
- 23:35PyArrow filesystem spec
- 25:25Alasio as super caching layer
- 27:10Q&A
related talks
27mHow to Eliminate the I/O Bottleneck and Continuously Feed the GPU While Training in the...
38mExploring Distributed Caching for Faster GPU Training with NVMe, GDS, and RDMA
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed Caching
24mHow to Eliminate the I-O Bottleneck & Continuously Feed the GPU While Training in the Cloud
31mGotta Cache 'em All: Scaling AI Workloads With Model Caching in a Hybrid...
05m⚡ Lightning Talk: Using Decentralized File System to Optimize ML/AI Workloads on Kubernetes
21mOptimizing Model Serving on Kubernetes With Model Streaming
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective