A\B Test Best Practices and Fun Case Studies from Deliveroo
summary
Deliveroo runs about 200 AB tests/year and uses experiments to turn guesses into dollars: removing the email button background increases sign‑ups, persisting baskets to disk raises completed orders, and preloading the home feed cuts time‑to‑interactive by 30% and drives more purchases. Adding images to recommendations lifts add‑to‑basket revenue (a later image bug cost measurable dollars and justified a hotfix), enabling tier switching increases Plus retention and often upsells users. Recommend feature flags for safety and Firebase recommendation for small teams.
AI-written summary · about
chapters (10)
- 0:00Introduction and setup
- 1:55Sign-up screen AB test
- 6:08State restoration in app
- 9:20Home feed loading optimization
- 13:33Bug hotfix decision with data
- 16:01Plus subscription tier switching
- 19:33Pop-ups impact on sign-ups
- 22:47Convincing leadership to AB test
- 31:04AB testing best practices
- 34:11Q&A and summary
related talks
55mHow to Improve Your Game's Retention & Monetization Through Data-Driven Experimentation
43mA/B Testing: What to Pay Attention To
45mA/B Testing From The Ground Up
28mMeasurement and Analysis for Sustainable Service Delivery
05mAlexander Shurkaev at RIT++ 2012
40mNext Best Action: Maximizing Customer Value
47m10K Metrics, 500 A/B Experiments and 500M p‑values Every Day — How Is That Possible?
26mOpen Source Warehouse Native A/B Testing.