How to train your general purpose document retriever model
summary
A learned sparse model based on SPLADE-style learned sparse model architecture is fine-tuned on ~1M queries (Stack Overflow titles + Google Natural Questions), distilled from cross-encoder and monoT5 teachers with histogram matching, and uses a FLOPS regularizer to spread token weights. It beats BM25 zero-shot across the BEIR suite (all but one dataset), encodes ~50 active query tokens and ~150 per doc from a ~30k vocab, and when combined with BM25 via reciprocal rank fusion gives consistent hybrid gains.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 0:32Overview
- 1:21Text retrieval paradigms
- 2:11Transformer models
- 3:53Problems withTransformer models
- 6:54Sparse models
- 8:22Benchmarks
- 9:42Merging
- 10:48Training pipeline
- 11:32Data sets
- 13:28Distillation
- 15:24Cross encoders
- 17:26Monotify
- 19:15Scaling
- 21:09Efficiency
- 22:50Batches
- 24:27Troubleshooting
- 28:43Conclusion
- 30:10Whats next
- 34:52Questions
related talks
45mHybrid search: Greater than the sum of its parts?
41mFine-tuning Neural Sparse Model for Domain Specific Data From an Existing OpenSearch Index
33mDeep research at scale
14mHow we taught agents to use good retrieval
32mBudget Friendly Semantic Search With Neural Sparse Search
01h 30mBuilding a Search Engine
01h 38mTraining Large Language Models: Practices and Research Questions
31mTaking AI/ML to the next level: Text!