Mobile Experimentation
summary
Instacart runs roulette, an assignment engine serving tens of thousands of experiments to millions of devices with billions of daily assignments, using deterministic XX hashing plus weighted rendezvous hash to support dynamic weights without reassigning users. Back end clients pull append-only feature defs; mobile shifts assignment to a centralized service so SDKs download tiny compressed evaluation blobs (often 95% compression) and persist them to disk, while full datasets live in an S3 cache for fast cold starts.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:09What is Instacart
- 2:53Fragmented experiment systems
- 3:56Roulette assignment goals
- 5:18Stateless deterministic assignments
- 7:01Randomized variant assignment
- 9:50Dynamic weights and hashing
- 11:13Roulette architecture
- 12:18Polling vs push updates
- 13:41Append-only database updates
- 15:28Caching experiment data
- 16:31Mobile experimentation challenges
- 18:40Recentralizing mobile SDK
- 20:04Minimizing mobile network access
- 21:28Current challenges and future
related talks
45mA/B Testing From The Ground Up
59mThe Future of automated testing
33mHow to A/B test with confidence
43mLean QA: Automation, Monitoring, Experiments
28mOptimizing your A/B testing roadmap
30mTesting Mobile Web Apps on Real Devices: How Fortune 100 companies do it
25mMake a Difference with Simple A/B Testing
55mModern automation approaches