From Inverted Index to Columnar Vectorized Execution Search
summary
Shows how Lucene/OpenSearch moves from per-document scoring to batch columnar paths: BKD range counting cuts an 8s monthly histogram to ~200–300ms (50x speedup), then multi-range traversal doubles that gain to over 100x in some cases; bulk docvalues + sparse 4K-block metadata yields 28x on an HTTP logs aggregation (998ms → 36ms) and up to 40% via bulk APIs. Execution uses SIMD/AVX-512, popcount tricks, and Apache Arrow Flight streaming; trade-offs include repeated key materialization and higher coordinator memory.
AI-written summary · about
chapters (14)
- 0:00Introduction and motivation
- 1:02Historical context and evolution
- 3:07Storage vs execution model
- 6:20Optimizing inverted index with BKD trees
- 8:45Multi-range traversal for aggregations
- 12:40Limitations of inverted index approach
- 14:27Columnar storage advantages
- 18:41Bulk collection and vectorization
- 22:36SIMD compute pipeline
- 25:03Inverted vs columnar trade-offs
- 26:50Cluster-level execution model
- 28:56Streaming aggregations with Arrow
- 32:06Future work and optimizations
- 33:53Q&A session
related talks
29mFrom Bottlenecks To Breakthroughs: Optimizing Aggregation in OpenSearch for Lightning-Fas...
19mVendor-Agnostic Analytics: Querying Open Columnar Formats With OpenSearch
45mAccelerating OpenSearch With Streaming: Apache Arrow, Flight, Data...
42mUsing OpenSearch as a Database With DataFusion
26mHigh-Performance Full-Text Search on Apache Iceberg Using Lucene-Integrate...
29mWide Event Analytics
21mReporting Systems Built on Top of Hadoop & Lucene
14mVector Search Benchmark[eting]