ARK: GPU-driven Code Execution for Distributed Deep Learning
summary
Claims a GPU-driven I/O path that lets GPU threads initiate DMA and run a single persistent GPU loop kernel so GPUs handle control and scheduling autonomously. Implements a software DMA proxy and an external FPGA DMA prototype that reaches line-rate from 8KB messages, reports 3.9x higher small-message throughput and 4.3μs latency (9.1x faster) versus CPU control, reduces I/O interference to raise all-reduce throughput 1.8x, and yields up to 2.1x training and 3.5x inference speedups using virtual CTA scheduling.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:23Modern Al Systems at Scale
- 0:44Communication Overhead in Distributed DL (1/2)
- 2:35Existing Systems: Overview
- 4:26ARK: A GPU-driven System - Key Design
- 5:32GPU-controlled DMA Design
- 6:04External DMA Engines
- 6:59DMA Engine Microbenchmark
- 7:33GPU-driven System Architecture
- 8:39Virtual CTA (Cooperative Thread Array)
- 10:19Code Generation & Compilation
- 11:11Evaluation: Data-parallel Training
- 11:42Evaluation: Tensor-parallel Inference
- 12:00Evaluation: Pipeline-parallel Training
related talks
26mDeepSpeed: Efficient Training Scalability for Deep Learning
16mGMI-DRL: Empowering Multi-GPU DRL with Adaptive-Grained Parallelism
18mGPU-Disaggregated Serving for Deep Learning Recommendation Models at Scale
01h 06mFoundation of GPU Programming
30mGPU Sharing at CERN: Cutting the Cake Without Losing a Slice!
38mExploring Distributed Caching for Faster GPU Training with NVMe, GDS, and RDMA
01h 53mMastering Agentic Coding and GPUs
09m⚡ Lightning Talk: Advanced GPU-Orchestrated Workflows and HPC Integrations on K8s for Distributed AI/ML at Scale