Interface Design Optimization as a Multi-Armed Bandit Problem
summary
Researchers apply UCB1 and a variant of upper confidence bounds to a live online math game with thousands of players and run meta-experiments (~10,000 subjects) comparing bandits to random assignment. Bandits find high-performing arms faster and produce more total engagement than random allocation, but a huge submarine condition surprisingly performs best under the chosen engagement metric, exposing a time-of-day confound and failure of pure sequential allocation. Recommendations: keep humans in the loop, inject randomization or use Thompson sampling, and reexamine optimization metrics for ethical deployment.
AI-written summary · about
chapters (13)
- 0:00Introduction to the talk
- 0:28Design spaces and optimization
- 3:19Battleship Numberline game demo
- 5:30Multi-Armed Bandits introduction
- 7:25Bandits vs random assignment
- 9:37Meta-experiment results
- 11:24Reducing designer workload
- 13:28Why bandits failed in practice
- 14:55Need for randomization
- 15:39Human oversight in automation
- 16:43Ethical implications of testing
- 18:10Future of AI-assisted design
- 18:55Q&A session
related talks
20mWinning the Game: Leveraging Contextual Bandits for Creative Optimization
48mIncentivizing Exploration among Behavioral Agents with Unbiased Histories
10mCooperative Design Optimization through Natural Language Interaction
11mThe Benchmarks Game: Why It's Rigged and How You Can (Really) Win
43mTaming the Unpredictable: Reliability in Chaos
20mAgentic Evaluations at Scale, For Everybody
32mAI Engineering Anti-patterns
44mAI Engineering Anti-patterns