Language Model Alignment: Theory & Algorithms
summary
Claims that best-of-N often matches or beats KL-regularized RL baselines and that a new inference method controlled decoding learns a prefix scorer to keep the base model frozen while solving the RL objective. Empirical tests use Bradley-Terry reward models and tokenwise value functions, show blockwise decoding cuts user latency and that surprisingly small KL budgets (around KL≈8 in chat settings) mark the safe regime, while distillation often loses best-of-N gains.
AI-written summary · about
chapters (14)
- 0:00Introduction to alignment
- 2:16Defining the alignment problem
- 4:13Best-of-n baseline
- 8:16KL-regularized reinforcement learning
- 13:39Visualizing alignment solutions
- 16:57Methods and algorithms
- 20:00Role of KL regularization
- 24:40Controlled decoding (YAAM)
- 28:40Value function training
- 34:06Blockwise control strategy
- 38:16Evaluation metrics
- 42:23Theoretical analysis
- 51:02Empirical results and multi-objective
- 56:31Conclusion and takeaways
related talks
45mOn Spurious Associations and LLM Alignment
19mRL for Autonomous Coding
42mRethinking AI Alignment and Evaluation under Heterogeneous Preferences
34mBeyond Scoring: Rethinking Ranking in the LLM Era
24mLLM Inference Optimization From Simple To Sophisticated
19mTraining Agentic Reasoners
01h 30mToward Understanding In-context Learning
40mIs my Large Language Model a Strange Loop?