Offline Ranking Validation - Predicting A/B Test Results
summary
Otto's search team builds an offline validator that predicts A/B outcomes by reweighting clicks with inverse propensity scoring and by correlating model NDCG with onsite KPIs. Two candidate rankers—order-only and click+order model—are evaluated on two weeks of logs using IPS-DCG and sum-of-relevant-ranks; the click+order model later yields about ~1% CTR uplift in a live A/B test, while conversion uplift is inconclusive. Key limitations: noisy low-propensity clicks, need for accurate propensity estimates, and failure on truly novel or long-tail placements.
AI-written summary · about
chapters (12)
- 0:00Introduction to talk
- 1:28About Otto company
- 3:14E-commerce search challenges
- 5:19Motivation for model ranking
- 6:01Full information ranking evaluation
- 9:30Implicit feedback and position bias
- 12:18Inverse propensity scoring
- 13:41Correlation with A/B test KPIs
- 15:45ABC test validation results
- 17:31Drawbacks and limitations
- 18:57Future plans and outlook
- 20:18Q&A session
related talks
35mLearning to Judge
25mBetter search relevance using Learning to Rank at mobile.de
42mA/B testing with real-world data to evaluate ML model updates
30mPredictive Analytics in Social Media and Online Display Advertising
31mEverything in its Right Place: Optimising Ranking in Online Grocery
41mHow Deliveroo improved the ranking of restaurants
30mWinning (Some) Churners Back
20mWinning the Game: Leveraging Contextual Bandits for Creative Optimization