Learning to Judge
summary
Otto builds ranking gold standards from customer logs by computing Bayesian-smoothed click probabilities per query and correcting for position bias with inverse-prop weighting, then re-ranks top 100 results in Solr (5000-query sample) to validate. A/B tests show +4% CTR / -3% conversion, traced to a 20–40% drop in product availability at top positions; filtering out delayed items and applying a filter products <100 views threshold improves metrics but still does not beat status quo.
AI-written summary · about
chapters (10)
- 0:00Introduction to learning to judge
- 1:35Learning to rank in a nutshell
- 5:08Defining relevance with judgments
- 9:43First experiment and surprising results
- 13:13Availability matters in ranking
- 16:44Including availability in judgments
- 20:15Judgment quality and data thresholds
- 23:53Continuous testing and learning approach
- 24:56Why testing judgments is crucial
- 26:39Q&A session
related talks
24mOffline Ranking Validation - Predicting A/B Test Results
47mLearning to rank search results
31mSponsored Session: From Clicks to Rankings: Building a Learning-to-Rank Pipeline with…
43mLearning to Rank for Reddit Search: A Project Retro
31mEverything in its Right Place: Optimising Ranking in Online Grocery
42mA Bayesian Approach to the Cold Start Problem
43m[Machine Learning] Creating more relevant search results with Learn to Rank
40mLevel Up: Everything you need to know about search, ...