GPU-Disaggregated Serving for Deep Learning Recommendation Models at Scale
summary
Prism disaggregates CPU and GPU pools via graph-level partitioning and RDMA so embedding lookups run on CPU nodes and dense nets on GPU nodes, using a fusgraph op, shared memory pool and GPU-direct RDMA for zero-copy transfers. Production results: deployed two years, runs on >10k GPUs, reduces CPU fragmentation 53% and GPU fragmentation 27%, enables capacity loaning from training clusters saving >90% GPUs during peaks, and adds at most ~6% latency in the worst case.
AI-written summary · about
chapters (13)
- 0:00GPU-disaggregated serving intro
- 2:03DRM resource provisioning challenges
- 4:33GPU cluster heterogeneity
- 5:59GPU disaggregation approaches
- 7:25Prism system design
- 8:50Disaggregated serving requirements
- 10:15Graph partitioning & RPC optimization
- 11:20Resource manager & scheduling
- 13:05RDMA communication scheduler
- 14:09Prism performance evaluation
- 14:50Resource reduction results
- 15:52Handling traffic spikes
- 16:15Future directions & takeaways
related talks
13mARK: GPU-driven Code Execution for Distributed Deep Learning
49mDistributed Deep Learning
15mDistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language...
14mP3: Distributed Deep Graph Learning at Scale
10mLightning Talk: Scaling Recommendation Systems To 2K GPUs and Beyond
14mFrom Reservation to 1,600 GPUs: A Petabyte-Scale Ray Data Pipeline
20mLegion: Automatically Pushing the Envelope of Multi-GPU System for Billion-Scale...
17mScaling Reinforcement Learning