Reproducibility in Embedding Benchmarks
summary
Shows concrete fixes and numbers: MTEB (and MMTEB/MIEB) adds task and model metadata, stratified subsampling and embedding caching to cut workload (tasks down from ~550 to ~130, documents to 2%), turning a ~2‑day/$1,800 run into ~3 hours/~$10. Demonstrates prompt sensitivity where a missing sentence flips scores, highlights smaller multilingual models matching 7B results, and gives a standardized model interface and practical implementation notes so reproduced results actually run, not just in principle.
AI-written summary · about
chapters (10)
- 0:00Embedding benchmarks introduction
- 3:57Why reproducibility in embedding benchmarks is hard
- 6:28Compute and cost barriers
- 10:21Stratified sampling optimization
- 15:06Multilingual benchmark assumptions
- 17:15Prompt impact on models
- 21:12Multimodal reproducibility challenges
- 24:27Standardizing model interfaces
- 26:37Why reproducibility matters
- 29:25Unsolved challenges
related talks
25mReproducibility in Embedding Benchmarks
19mHow to look at your data
29mModel Evaluation for Custom Datasets With Open Models: Multi-Model Comparison With...
30mPanel Discussion: Are We Measuring Intelligence or Just Benchmarks?
31mChallenges and Considerations in Language Model Evaluation
01h 30mBuilding a Search Engine
01h 08mGoing multi-modal
13mOptimizing and Benchmarking Open-Source LLMs