Leveraging Lucene Skip Indexes for Faster Aggregations in OpenSearch
summary
Shows that Lucene 10 skip index metadata for sorted doc values lets OpenSearch skip or bulk-count whole doc ranges instead of visiting each document, aligning filter doc iterators with skip-intervals and using popcount bitset ops for block-level bucket counting. Prototype reduces hourly-with-filter latency from ~1s to 36ms (~28x), yields ~40% speedup with metric sub-aggregations and up to 6x on some date-range tests, costs <0.1% extra storage per timestamp field, works best on sorted time-series, and must be enabled at index creation.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:07Filter rewrite recap
- 7:10Filter rewrite limits
- 12:16Lucene skip indexes intro
- 14:08Skip indexes in Lucene 10
- 17:17Skip indexes for aggregations
- 20:50When skip indexes help
- 22:13Skip indexes benchmarks
- 23:40Optimization integration
- 24:48Enabling skip indexes
- 26:11Community improvements
- 27:16Future JVM optimizations
- 30:27Skip index resources
- 31:15Q&A
- 33:28Q&A
related talks
30mCaching for Speed: Practical Performance Optimization in OpenSearch & Lucene
30mBoosting OpenSearch Performance: Lucene Bulk Collection and gRPC Search in Action
41mConstant-Time Aggregations with Star-Tree in OpenSearch
37mDerived Source: Slash Storage Costs Without Losing Data in Open...
18mSpeed Up Your Search: Concurrency and Merge Policies to the Rescue
23mBeyond Slow Logs: Implementing Advanced Query Insights in OpenSearch
36mSearch on the fly: how to lighten your Big Data
37mAlgorithms & data-structures that power Lucene & ElasticSearch