Document classification search; joins vs payloads
summary
Adds inequality ops to the payload check query parser (see LUCENE-9659 / SOLR-14787) and encodes ML classification confidences as payloads so queries can test gt/lt at term positions. Reference implementation indexes VGG16/YOLO outputs for COCO (about 1,000,000 docs with avg 50 classifications in benchmarks and a 118k-image demo). Benchmarks: index-time filter indexes ~11k docs/s, per-field ~209 docs/s and huge memory, joins balloon index and are very slow, payloads cut indexing and query cost dramatically (≈250 q/s) while keeping flexible thresholds.
AI-written summary · about
chapters (13)
- 0:00Introduction and overview
- 1:57What are payloads in Lucene
- 4:43Use case motivation
- 7:31Approach one filtering at index time
- 9:15Approach two per-field labels
- 11:01Approach three join queries
- 14:10Approach four payloads extension
- 16:14Benchmarking setup
- 17:38Indexing performance results
- 20:30Query performance results
- 22:18Live demonstration
- 25:17Future enhancements
- 27:02Q&A
related talks
34mClassifying Search Queries Without User Click Data
22mImplementing a Text Classifier based on Lucene & LIBSVM
37mLexical, Vector & Hybrid Search with Elasticsearch
46mNatural Language & Search, Improve your Search w/ Vector capabilities
41mHow to train your general purpose document retriever model
44mContent Classification: NLP vs. AI in the Age of Data Vectorization
20mText Search on Images with Quantized ColPali
01h 30mBuilding a Search Engine