Custom Pipeline Parallelism for Frontier-Scale Model Serving
summary
vLLM’s compiled-DAG deadlocks beyond two pipeline stages, so the team replaces it with a custom Ray executor that uses actor mailboxes as the scheduler and torch send/receive over Ray to pass activations. Two hidden CPU bottlenecks are removed: large pickled tensor metadata is replaced with a small metadata index table sent over nickel, and the costly GPU->CPU .item sync is avoided via a speculative receive based on scheduler token-size hints, yielding up to 1.5x throughput.
AI-written summary · about
chapters (10)
- 0:00Introduction by hosts
- 0:26Overview of PTT architecture
- 2:12Challenges with VLM assumptions
- 3:17End-to-end request flow
- 4:00Optimizing the fast path
- 5:25Debugging Ray compile DAG deadlocks
- 8:00Solving CPU transfer bottlenecks
- 13:03Eliminating GPU-CPU sync overhead
- 16:35Benchmarking improvements
- 19:28Key takeaways and lessons learned
related talks
16mRevisiting Pipeline Parallelism for LLM Serving
10mGraph Based Pipeline Parallelism
21mMulti Runtime Serving Pipelines
11mvLLM & Ray Updates
31mThe Evolution of Multi-GPU Inference in vLLM
17mAttack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
29mScaling ML Pipelines for Autonomous Driving with Ray