Building a Search Engine
summary
Demo builds a working search stack: index 25k Unsplash captions into BM25 on Lucene (Elasticsearch) at about 5000+ docs/sec, then compare against embeddings from sentence-transformers MiniLM and CLIP hosted on Hugging Face. Embedding retrieval yields qualitatively better matches for many queries but raises latency (Elasticsearch hits ≈3 ms, naive embedding matrix search ≈200 ms) and cost, so workshop shows FAISS/IVF + nprobe ANN to scale recall; notebooks and code on the provided GitHub reproduce the experiments and tradeoffs.
AI-written summary · about
chapters (17)
- 0:00Building a search engine
- 1:27Workshop setup
- 5:00Dataset and traffic
- 9:38Token-based retrieval
- 15:08BM25 algorithm
- 19:34Elasticsearch implementation
- 36:45Token retrieval limits
- 39:54Embedding-based retrieval
- 48:06How embeddings work
- 53:25Sentence Transformers
- 1:05:28Embedding performance
- 1:06:32CLIP multimodal embeddings
- 1:13:51Vector databases scaling
- 1:19:53Approximate nearest neighbor
- 1:21:17Token vs embedding comparison
- 1:24:31Production search stack
- 1:27:45Q&A
related talks
01h 28mBuilding a Semantic Search Engine
30mBeyond Keywords: Building an AI-Powered Search That Reads Your Mind
38mNatural Language Search-Lexical and Semantic
32mRetrieval Augmentation and Semantic Search at Scale
22mThe Capabilities of Semantic Search
26mAdvancing Multi-Modal Search Capabilities in Search Pipeline
19mVectorize Your Open Source Search Engine
47mThe Neural Search Frontier #bbuzz