PUZZLE: Efficiently Aligning Large Language Models through Light-Weight Context...
summary
PUZZLE implements a two-dimensional model-orchestration system that reduces context-switch overhead during RLHF by adding time-sharing within stages and a workload-aware switching planner across stages. It runs on clusters up to 32 GPUs, compares to DeepSpeed-Chat, and reports up to 2.12× end-to-end speedup (average 1.8× on PCIe, 1.5× on NVLink). Microresults show the time-sharing plan yields ~1.12× intra-stage gain and workload-aware transformations deliver ~1.22×–1.34× inter-stage gains.
AI-written summary · about
chapters (9)
- 0:00PUZZLE and alignment challenge
- 1:03Reinforcement learning from human feedback
- 3:30Proximal policy optimization workflow
- 5:21Context switching challenges in alignment
- 8:14PUZZLE’s interstage context switching
- 10:42Time-sharing strategy for optimization
- 11:03Workload-aware switching for transitions
- 12:47PUZZLE evaluation results and gains
- 14:36Conclusion and future directions
related talks
13mOptimizing RLHF Training for Large Language Models with Stage Fusion
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
14mScalable Solutions for Running Large Language Models
32mEnd-to-End Optimizing Multi-Turn RL and High-Performance Inference in Agents with...
19mDynaRL: Flexible and Dynamic Scheduling of Large-Scale Reinforcement Learning Training
15mResource Multiplexing in Tuning and Serving Large Language Models
36mSkyRL: Democratizing RL Training for Researchers
39mAccelerating Open-Source RL and Agentic Inference with vLLM