Word2Vec model to generate synonyms on the fly in Apache Lucene
summary
Implemented an on-the-fly Word2Vec synonym filter for Apache Lucene that reads a trained deep learning4j model, builds an HNSW proximity graph from embeddings and expands queries or enriches documents. Prototype reduces per-term expansion latency from 70ms→6ms, parses DL4J zip models only, and includes a CLI trainer that builds a model from a Lucene index (tested on a 3.4 GB Italian Wikipedia index); HNSW build for ~300k vectors takes ~2 minutes and current work targets on-disk memory-mapped models and multi-term synonyms.
AI-written summary · about
chapters (19)
- 0:00Introduction and presenters
- 1:25Company background and expertise
- 3:12Agenda and talk overview
- 3:54Synonym expansion in search engines
- 6:25Limitations of vocabulary-based synonyms
- 7:28Proposed Word2Vec solution
- 8:31Word2Vec algorithm overview
- 10:38Word2Vec architectures
- 12:23DeepLearning4j implementation
- 14:11Contribution to Apache Lucene
- 17:43HNSW for efficient nearest neighbor search
- 20:11Implementation details and optimizations
- 21:35Performance improvements
- 23:44Contextual domain adaptation
- 24:52Word2Vec model trainer tool
- 26:57Practical usage examples
- 29:27Index-time vs query-time expansion
- 31:55Future work and improvements
- 35:04Q&A session
related talks
37mLexical, Vector & Hybrid Search with Elasticsearch
41mSearching for the Right Words: Bringing NLP to Apache solr through ONNX and OpenNLP
19mVectorize Your Open Source Search Engine
42mResearch in the Age of AI
45mDeep Learning and NLP with Spark
32mBuilding semantic search using Deep Learning
37mLike2Vec with neural network embeddings
01h 01mWord Embeddings - An Alternative & Efficient Approach to Search for Docs