Introducing Multi-valued Vector Fields in Apache Lucene
summary
Adds multi-valued KNN vectors to Lucene by making each HNSW node a vector ID and building an ordinal map to documents so the index can map ordinal vector ID to document ID using a docsWithVectorSet accumulator and a direct monotonic writer; query side modifies the neighbor queue to aggregate per-document scores with max or sum aggregation (max preferred initially), supports float and byte vectors, ships as a working prototype PR for Lucene 9.x, reduces the patch from 85 to 25 classes, and flags performance and length-bias issues for further benchmarking.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:23About us
- 3:05Motivation
- 4:34Considerations
- 7:18Multivalued Vector Fields
- 17:05Query Time
- 20:45Lucene
- 22:42Indexing Time
- 27:53Exact Search
- 29:11Approximate Search
- 29:57Enable Queue
- 31:52Challenges
- 33:24Wrapup
related talks
42mApache Lucene Vector Search Update
52mBeyond Single Vectors: Multi-Vector Search for Enhanced...
39mWhat's in a Vector Database?
45mRevolutionizing Search with Power of Neural Sparse and Dense Vectors
47mBoosting AI With OpenSearch: Vector Stores, Similarity Search, and Retrieval-Augme...
45mEnd-to-End Semantic Search with Apache Solr 9.8 LLM Module
41mKeynote: From Keyword Search to Vector Powered Insights: Th...
44mVector Search Isn’t Enough