Implementing a Search Index
summary
Describes building a Go-based search index that converts text to vectors (using pretrained CSV word2vec models and TensorFlow Universal Sentence Encoder), stores vectors in a nearest-neighbour structure (KD/BP-style tree) and ranks by distance. Tests on BBC News (2000–2004) compare Elasticsearch keyword hits vs average word2vec vectors vs sentence encoder: vector methods return semantically relevant results (e.g. answers about concepts not literally present) but pretrained models miss domain terms, so training word2vec on your corpus fixes that; overall the approach works and improves relevance.
AI-written summary · about
chapters (27)
- 0:00Introduction
- 0:36Machine Box
- 1:03Search
- 1:42Why Search
- 2:02Reduce Cost
- 2:29Makes Users Happy
- 2:45Search Engines
- 3:43Lucky Search
- 4:10Generic Search
- 5:12Google
- 5:35Quick Auto Response
- 6:17Two Simple Steps
- 6:35Machine Learning
- 7:23War2 Vector
- 10:04Alberene
- 10:58CSV
- 11:38Vector Representation
- 13:01Tokenization
- 13:47Transformation
- 13:57Vector
- 15:52BB Tree
- 17:00Demo
- 18:27What is Say
- 19:38Elastic Search
- 20:54Universal Sentence Encoder
- 22:23Training
- 24:20Conclusion
related talks
59mContextual search with vector search: exploring your options with open source tools
01h 01mWord Embeddings - An Alternative & Efficient Approach to Search for Docs
19mVectorize Your Open Source Search Engine
01h 09mVector Search: Ask Me Anything!
31mBuild your own search application with Weaviate
16mDense Vectors in Elasticsearch
25mBeyond Filters: Modern Search with Vectors in Django with Kumar Shivendu
32mBuilding semantic search using Deep Learning