Reproducibility in Embedding Benchmarks
summary
MTEB pins dataset and checkpoint revisions and version-controls code, but reproducibility breaks from compute costs, prompt sensitivity, and implementation quirks. Practical fixes include 4% sampling with 10 resamples and k‑means for clustering, 1k-query sampling with top‑250 docs capped at 250k for retrieval, caching repeated parallel sentences, and pruning ~500→130 tasks. Result: English runs drop to ~3 hours on an H100 (~$10) versus days on P4d, and multilingual tests show a 12x smaller model matching a 7B model.
AI-written summary · about
chapters (20)
- 0:00Introduction to MTeb
- 1:46Reproducibility challenges in benchmarks
- 3:09Compute costs of benchmarking
- 5:12Balancing benchmark coverage and compute
- 5:55Speeding up benchmarks with sampling
- 7:20Filtering tasks for multilingual benchmarks
- 8:25Reducing runtime with smaller models
- 9:27Multilingual model performance gaps
- 10:10Prompt sensitivity in instruction-tuned models
- 11:57Tracking prompts in MTeb
- 12:41Reproducibility in multimodal embeddings
- 14:49Standardizing model interfaces
- 15:52Efficiency in embedding models
- 16:33Benefits of using MTeb
- 17:37Unsolved reproducibility issues
- 18:41Future of data set hosting
- 19:02Q&A: Redefining tests
- 20:06Q&A: Adding new data sets
- 21:31Q&A: Re-evaluating models
- 22:13Q&A: Data set contamination
related talks
32mReproducibility in Embedding Benchmarks
19mHow to look at your data
30mPanel Discussion: Are We Measuring Intelligence or Just Benchmarks?
01h 57mResearch Reproducibility in Embedded Learning
11mSotabench for Reproducible Research
29mModel Evaluation for Custom Datasets With Open Models: Multi-Model Comparison With...
18mSparse Checkpointing for Fast and Reliable MoE Training
10mKeynote: Scaling Open Source Large Language Models