Vectorize Your Open Source Search Engine
summary
Shows how to add vectors into an existing Lucene/Elasticsearch pipeline using off‑the‑shelf Hugging Face models, a lightweight encoding service, and Chorus plus Quirky for query rewriting and experimentation with Cupid for A/B metrics. Uses MiniLM (sentence‑transformers) for text and OpenAI CLIP (77‑token limit) for images, notes MiniLM handles >256 tokens. Reports BM25 still best for exact matches, text vectors excel on multilingual and intent cases, image vectors on visual features, and a hybrid BM25+vector approach (with occasional fine‑tuning) gives the most practical gains.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:12Vector search overview
- 2:52Search challenges
- 5:25Vectorization formula
- 6:52Implementation strategy
- 8:46Technical workflow
- 11:02Demo results
- 11:55Evaluation and results
- 13:56Implementation caveats
- 15:12Model fine-tuning and Q&A
related talks
59mContextual search with vector search: exploring your options with open source tools
37mLexical, Vector & Hybrid Search with Elasticsearch
46mNatural Language & Search, Improve your Search w/ Vector capabilities
01h 30mBuilding a Search Engine
22mVector Search Made Simple: Getting Started With OpenSearch for AI Applications
24mVector Search at Uber
01h 09mVector Search: Ask Me Anything!
14mSearch on Steroids