Lightning Talk: TorchRL - RLHF Support
summary
Adds RLHF primitives to TorchRL that let teams assemble training pipelines (model pretraining, reward-model training, RL fine-tuning) instead of using a single black‑box trainer, with concrete integrations for Hugging Face models and datasets. Implements TensorDict for nested parameter and batch handling, supports TorchRL-HF tooling, exposes standard losses including PPO and a K‑reward constraint, and offers memory-mapped parameters for partial model loading. Prototype is working end‑to‑end with examples but needs broader scale testing.
AI-written summary · about
chapters (8)
- 0:00RLHF and TorchRL intro
- 2:27How RLHF works
- 5:59TorchRL for RLHF
- 7:44TorchRL core features
- 9:12TensorDict in TorchRL
- 12:02RLHF fine-tuning tools
- 12:45RLHF research state
- 13:53Q&A
related talks
11mTorchRL: The Reinforcement Learning and Control library for PyTorch
21mAdvanced Ray RLlib: New Features and New Possibilities
05mWhat's New in the Torch Ecosystem
14mRLlib: Scalable RL for TensorFlow, PyTorch, and Beyond
13mLightning Talk: From Hugging Face To Handheld: Scaling LLM Deployment W...
11mSponsored Session: Lightning Talk: PyTorch and Democratization of AI Accelerators
13mOptimizing RLHF Training for Large Language Models with Stage Fusion
27mControlling the Chaos: Reproducible Deep Learning Experiments Using Lightning and Hydra