Efficient Scoring in Lucene
summary
Describes a concrete scoring algorithm built on Lucene primitives that implements early-termination / max-score style pruning over inverted lists, with code that hooks into Lucene’s posting iteration and scoring interfaces. Evaluated on a 2.5M English Wikipedia collection, it reduces the number of documents examined from ~7M to ~400k for typical queries and brings median response down into the ~150 ms range. Implementation is straightforward to add to an existing index and yields consistent, practically significant cost and latency reductions.
AI-written summary · about
chapters (7)
- 0:00Introduction to Lucene scoring
- 3:05Benchmarking with Wikipedia
- 6:09Scoring function optimization
- 9:17Inverted list performance
- 12:00Speed improvements in Lucene
- 15:24Proxy manager solution
- 18:15Q&A and closing
related talks
30mCaching for Speed: Practical Performance Optimization in OpenSearch & Lucene
37mAlgorithms & data-structures that power Lucene & ElasticSearch
40mWhat's evolving in Elasticsearch and Lucene?
40mLeveraging Lucene Skip Indexes for Faster Aggregations in OpenSearch
46mFull-Text Search Internals with Elasticsearch
41mPainless / Painful? A Custom Ranking System in Elasticsearch
25mApache Lucene: From Text Indexing to Artificial Intelligence
31mUnspoken overhead of the Inverted Index