Information Retrieval from the Ground Up
summary
Search sits under RAG as the retrieval layer and the demo shows concrete tradeoffs: Elasticsearch uses an inverted index with BM25 TF‑IDF and token analysis for fast lexical matches, adds OpenAI 128‑dim dense embeddings and a SPlADE‑style sparse model, and combines them into hybrid search with chunking, top‑k rescoring and reciprocal rank fusion. HNSW recalls vectors but incurs costly merges; engrams and fuzziness help typos but inflate index size. A shared demo instance (elastai.engineer) runs the examples and works end to end.
AI-written summary · about
chapters (28)
- 0:00Introduction to retrieval
- 3:44Defining the search problem
- 6:42Analyzing search tokens
- 10:22Stemming and stop words
- 13:38The role of stop words
- 16:10Indexing and data structure
- 21:26Inverted index mechanics
- 23:28Advanced lexical searches
- 25:54Stemming limitations
- 28:41N-gram tokenization
- 30:18Edge N-gram strategy
- 33:29Multilingual search
- 36:03Fuzziness and Levenshtein
- 38:37Scoring and BM25
- 41:42Relevance and coordination
- 44:28Interpreting scores
- 46:54Dense vector search
- 50:35Sparse vs dense embeddings
- 53:56Hybrid search approach
- 57:47Thresholds and quality
- 1:02:07Scaling and trade-offs
- 1:07:57RAG and query rewriting
- 1:11:10Semantic retrieval tactics
- 1:17:20Chunking and highlighting
- 1:20:31Hybrid search architecture
- 1:23:46Ranking and fusion
- 1:27:11Technology stack decisions
- 1:33:04Advanced reranking strategies
related talks
42mResearch in the Age of AI
30mSearch on Steroids — A Story of Semantics
34mImplementing an open-source RAG with OpenSearch
14mSearch on Steroids
02h 47mFrom naive to advanced RAG: the complete guide
46mFrom Naive Sorting to Natural Language Processing, 3 Years of Optimization of Our Search Engine
02h 51mBuilding Your Intelligent Assistant with Hugging Face and Elasticsearch
43mHarnessing AI to strengthen trustworthy information