Aligning ML objectives with human values
summary
Frames alignment as a reduction: start with a black‑box optimizer that gets low regret on a model class and aim to compete with that class while the true utility U is unobserved. Uses reward model via human queries and a recursive decomposition (ask humans to break hard evaluations into easier subquestions, use the current model to answer subtasks). Builds toy prototypes (transformer learners, simulated oracle/human decompositions) and finds on simple algorithmic domains that decomposition‑based supervised training learns almost as fast as having ground‑truth labels, but scalability and error compounding remain open.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:32The problem
- 4:22Model of goal
- 5:53Gaps
- 7:16Using humans as ground truth
- 11:28Challenge 1: humans are expensive
- 16:12"Semi-supervised" RL
- 17:30Simple approach
- 21:12Challenge 2: want to outperform humans
- 22:32Possible approach: decomposition
- 27:00Example
- 31:42Training process
- 39:54Simple prototype
- 40:14Simple domains
- 43:24Performance
- 43:43Next steps
related talks
42mRethinking AI Alignment and Evaluation under Heterogeneous Preferences
44mThe ML Mindset for Managers
30mAdmiral-Driven ML Framework for Marine Operations
13mHuman-in-the-Loop MLOps: Boost Model Performance by 40%
34mOffline Reinforcement Learning and Model-Based Optimization
28mPreference Learning from Minimal Human Feedback for Interactive
56mDehumanizing Agents: Why Explainability is Crucial in the LLM Era
44mAI Engineering Anti-patterns