Supercharging your transformers with synthetic query generation and lexical search
summary
A 1.5B-parameter GPT-style model is fine-tuned on 650k MSMarco/Natural Questions pairs to generate 20 synthetic queries per document, producing a synthetic corpus used to train a dense encoder with contrastive loss. The fine-tuned retriever is combined with OpenSearch BM25 (scores normalized and fused, geometric mean works best) and yields a 14.30% nDCG@10 boost versus BM25 across 10 public test sets. Query generation for 250k docs costs under ~$500 and fine-tuning under ~$100.
AI-written summary · about
chapters (10)
- 0:00Introduction to the talk
- 2:07Problem with enterprise search
- 8:27Solution: Fine-tuned models with BM25
- 12:00Synthetic query generation technique
- 20:06Choosing synthetic query generator
- 27:31Training the dense encoder model
- 33:10Combining BM25 with fine-tuned models
- 34:34Experimental results and improvements
- 36:18Future directions and open questions
- 42:46Q&A session
related talks
41mFine-tuning Neural Sparse Model for Domain Specific Data from...
41mFine-tuning Neural Sparse Model for Domain Specific Data From an Existing OpenSearch Index
20mGenAI synthetic data used to fine tune a domain specific model
44mRediscover your keyword search: Expand, Enrich and Rewrite
21mFrom Data to Precision: Fine-Tuning LLMs for Proprietary Query Languages
49mBen Hall & Barbara Fusinska
17mUnlock the power of Generative AI: From understanding its principles to utilizing its power
01h 30mBuilding a Search Engine