Improving Search @scale with efficient query experimentation
summary
A production “Query Testing” platform routes traffic to experiment on a query-level randomization unit (not users/sessions), auto-identifies candidates, aggregates queries by intent, and uses group sequential testing plus prior-informed blending to stop 60% of doomed runs early and cap experiments at 28 days. 1,900 experiments run so far show variants beat baseline in ~8.5% of cases with an average treatment effect of 16.1%; metrics are CTR, add-to-basket rate and average revenue per query, and the system works across any search engine.
AI-written summary · about
chapters (8)
- 0:00Introduction and approach
- 2:24The challenge of search scale
- 7:47Experimentation mechanics
- 12:55Defining success metrics
- 15:47Designing the platform
- 21:22Overcoming data obstacles
- 32:56Results and communication
- 37:23Q&A and closing
related talks
55mHow to Improve Your Game's Retention & Monetization Through Data-Driven Experimentation
47m10K Metrics, 500 A/B Experiments and 500M p‑values Every Day — How Is That Possible?
34mBuilding a WebScale Search Engine with a Data Driven Architecture
14mVector Search Benchmark[eting]
45mA/B Testing From The Ground Up
31mBuilding a Data Experimentation Platform Without a Platform Team
22mHead-N-Tail Analysis to Increase Engagement #bbuzz
35mThings I Learned the Hard Way Building a Search Engine