No GPU Left Behind: Scaling Online LLM Training With...
summary
Implements co-located vLLM inside TRL so inference runs in-process (no HTTP server), removes ping-pong GPU idle time, and supports torchrun SPMD with DP and TP (FSDP/DeepSpeed). A merged PR ships an automation to pick GPU memory utilization; experiments scale to 72B models and show up to 1.7× throughput (community reports up to 3×), fewer GPUs needed, equivalent training/reward curves, and practical use of vLLM sleep mode and offloading to fit larger batches.
AI-written summary · about
chapters (9)
- 0:00Talk overview
- 1:04GRPO workflow
- 3:33Inference challenges in GRPO
- 4:57Collocated training and inference
- 6:44Parallelization implementation
- 12:18Experimental results
- 14:02Highlights and feedback
- 14:47Takeaways and best practices
- 16:20Q&A
related talks
31mThe Evolution of Multi-GPU Inference in vLLM
20mFlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
26mEfficiently serving LLMs at scale.
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
08mHacks to Make LLM Training Faster
01h 18mOptimizing LLM Training and Inference Performance on GPUs
16mAccelerating Design Space Exploration for LLM Training Systems with Multi-experiment...
33mAccelerating the LLM Lifecycle on the Cloud