Building Learning to Rank models for search using LLMs
summary
LLMs generate binary and listwise relevance judgments (Anthropic Cloud prompts) that train supervised LTR models (pointwise regression, ranknet, LambdaRank, LambdaMART) using 61 hand‑crafted features plus vector cosine and knowledge‑graph overlaps; evaluation uses mean average precision at 10 on ElasticSearch top‑50 results and a Quadrant vector store. RankNet performs best and the ML reranker approaches a long‑tuned pipeline (≈20% precision lift claimed); LLM labels are ~70% concordant with humans and tend to be more lenient so active learning or ensembling is proposed.
AI-written summary · about
chapters (19)
- 0:00LTR with LLMs intro
- 1:24Project overview
- 2:06Core LTR idea
- 4:34Baseline pipelines
- 5:39System architecture
- 7:02Query sampling
- 8:09Label generation
- 11:19Feature engineering
- 15:12LTR model types
- 16:36Model performance
- 17:18Pointwise regression
- 18:00Pairwise models
- 20:49Rank Matrix
- 21:51Lambda Rank & LambdaMart
- 22:33Evaluating LTR models
- 24:19LTR with LLMs benefits
- 25:43Error analysis
- 27:09Future improvements
- 28:12Conclusion & Q&A
related talks
24mLearning "Learning to Rank"
47mLearning to rank search results
34mBeyond Scoring: Rethinking Ranking in the LLM Era
30mLLM-Based Recommendation Systems
49mLLM—Search Engine Killer?
55mLearning to Rank, a Little
33mFrom LLM-as-a-Judge To Human-in-the-Loop: Rethinking Evaluat...
43m[Machine Learning] Creating more relevant search results with Learn to Rank