Designing and Building Custom Reinforcement Learning Environments for Fine-tuning LLMs
summary
Built a token-level RL environment called Wikipedia Maze that casts pages as states and link clicks as actions, with deterministic terminals and a shaped reward: -1 for selecting a non-present target URL, -5 for gibberish, 0.5 for checkpoint links and 0.1 for reaching the goal. Uses the Flight orchestrator and an early AI gym library for episodes, runs rollouts with Olama and Gemma-3/12B models, explores vLLM for efficient trajectories, and achieves consistent small-scale 2-hop solves while flagging context length, prompt design, and reward engineering as bottlenecks.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:49RLHF/RLVR training bottlenecks
- 3:58Static datasets limitations
- 6:03Next token prediction inspiration
- 7:49Wikipedia as RL environment
- 10:38Wikipedia navigation agent design
- 13:08Implementation stack
- 14:11AI Gym library examples
- 16:47Multi-node setup
- 17:30Initial experiments
- 18:11Implementation challenges
- 20:40Debugging with flight
- 21:43Research roadmap
- 23:07Generalizing RL environments
- 23:53Related projects
- 24:16Q&A
related talks
39mLet LLMs Wander: Engineering RL Environments
15mAgentic Reasonings with Reinforcement Learning
41mReinforcement Learning and Minecraft
19mTraining Agentic Reasoners
41mLet LLMs Wander: Engineering RL Environments
35mLLM-Guided Reinforcement Learning for Mastery Learning: Large-Scale Field Evidence
27mBuilding Mini Minds Our Adventures in Teaching Small LLMs to Reason
19mRL for Autonomous Coding