Standing Up for the Unmeasurable in The Design Process
summary
Argues that blindly optimizing measurable metrics causes harm, citing Stockholm ambulance data showing 1.44% higher mortality under private performance targets and Google experiments that add 100–400 ms slowdown which cut searches per user by 0.2–0.6. Shows concrete anti-patterns: adding a banner or autoplay that raises sales or engagement but risks lowering retention. Recommends tools like targeted A/B tests, testing for elasticity, beta signups and deliberate regressions to learn tradeoffs rather than just hit OKRs.
AI-written summary · about
chapters (30)
- 0:00Introduction
- 1:06Data-driven design limits
- 3:11Performance metrics consequences
- 6:19Behavior vs design measurement
- 7:02Banner trap in apps
- 10:12Autoplay and engagement
- 11:58Asking why and cost
- 13:04Measuring better and guessing
- 15:29Good Judgment study
- 21:13Delayed judgment thinking
- 22:18Testing elasticity and learning
- 24:48Exploring problem space
- 26:56ML and design parallels
- 28:43Lean startup pitfalls
- 30:33Balancing measurable goals
- 32:22Question and answer session begins
- 32:43Convincing marketing on banners
- 33:47Elasticity testing reliability
- 34:53Unmeasurable qualities examples
- 35:37Explaining values to leadership
- 35:58Innovation success indicators
- 38:03Critique of happiness metrics
- 39:07Interpolating elasticity results
- 39:49Biggest design assumption wrong
- 41:34Deciding unmeasurable
- 43:20Proving impact without metrics
- 44:46Spotify Slack designs opinion
- 45:51UX research literature
- 47:14Research tech toolkit
- 48:18Closing remarks
related talks
56mEssential Analytics and Coding for the ‘Complete’ UX Professional
40mPanel Discussion: What Users Really Want
38mResponsible product design
36mPreventing Random Acts of Metrics
46mDesigning for impact
39mExperimentation and the gold standard of data champions
53mYour Attention, Please: Optimizing Metrics for Humans
22mA User Guide to Product Design