Vector Search Benchmark[eting]
summary
Marketing charts often hide the setup: vendors cherry‑pick a scenario to claim 40% faster overall by promoting one tuned case, and a single edge optimization can show 10x on one query while everything else is similar. Key pitfalls: most benchmarks use read‑only datasets that miss real read/write ratios, and HNSW filtering slows queries unless specially optimized. Elastic runs automated nightly benchmarks with Rally to catch slow regressions; do your own reproducible tracks measuring precision/recall, hardware, and read/write mix.
AI-written summary · about
chapters (7)
- 0:00Introduction to benchmarking
- 1:49Common benchmarking pitfalls
- 5:42Biases and defaults in benchmarks
- 8:11Cheating and glossy charts
- 9:57Making benchmarks meaningful
- 11:21Do your own benchmarks
- 13:03Learning from flawed benchmarks
related talks
33mBeyond Benchmarks: Real-World AI Search Performance in OpenSearch (Lucene Vs Faiss Vs JVector)
35mImpact of Vector Search
47mWhy, What, When and How of Running Diff Types of Sear...
22mComparing vector implementations in generic databases
42mImproving Search @scale with efficient query experimentation
39mMastering OpenSearch Benchmark: Advanced Techniques for Power Users
32mUnder the Hood: Key Algorithms Powering Vector Search
37mFrom Inverted Index to Columnar Vectorized Execution Search