Boosting Ranking Performance with Minimal Supervision
summary
With just three labeled queries and documents, an LLM (including open-source 3B models or GPT-style APIs) generates synthetic questions for every document, then a much smaller Transformer ranker is trained on that data; evaluation shows a ~10 nDCG@10 points lift over a strong BM25 baseline (example: TrackCOVID/Bio subset rises from ~69 to ~79). The pipeline uses a BM25 consistency check to filter low-quality generations, samples random and hard negatives for training, runs generation offline, and deploys the learned ranker and generated query suggestions in Vespa.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 0:47Ranking models
- 7:30Training Ranking Models
- 11:24Interaction Data Bias
- 12:07Cold Start
- 12:58Training Data
- 13:10Hot Takes
- 14:02HighLevel Overview
- 16:07Prompt Engineering
- 17:20Prompt Instruction
- 18:31Consistency checking
- 19:46Sample irrelevant documents
- 20:38Train ranking models
- 21:50Train ranking models in practice
- 23:46Large language models
- 25:17Query suggestions
- 26:13Language models
- 27:44Conclusion
- 28:16Questions
- 31:49Challenges
- 32:50Filter out synthetic queries
- 34:26Fine tuning
- 35:40Use cases
- 37:11Qualitative insights
- 39:27Other modes
related talks
29mBuilding Learning to Rank models for search using LLMs
34mBeyond Scoring: Rethinking Ranking in the LLM Era
22m360Brew: LLM-based Personalized Ranking and Recommendation
43mLarge Language Models for Devs: from zero to your first LLM application
01h 38mTraining Large Language Models: Practices and Research Questions
01h 05mControlling distribution shifts in language models: a data-centric approach.
33mDeep research at scale
35mLeveraging LLM for text augmentation in Topic Modeling