Accelerating Aggregations with Lucene-Aware Execution in OpenSearch
summary
Claim: replacing per-document JVM-heavy collectors with Lucene bulk primitives plus OpenSearch batching reduces aggregation CPU overhead dramatically. Implementation uses Lucene collect doc ID stream and intoArray to fill a 256-int buffer, a long values fast path to load values, and skip-indexed 4k blocks with popcount; date-histogram cases see up to 20x speedups (filter-rewrite earlier gave 50x, multi-range 100x), storage cost ~0.1% for timestamp; works well for time-series, fails on unsorted or multi-valued fields.
AI-written summary · about
chapters (13)
- 0:00Aggregation acceleration intro
- 1:23Slow aggregations problem
- 3:13Default execution flow
- 6:27Prior optimizations recap
- 8:53Lucene bulk APIs
- 14:34Batch collection design
- 21:16Skip indices integration
- 25:12Optimization limitations
- 26:38Automatic complementarity
- 27:22Future roadmap
- 32:48Key takeaways
- 34:53Resources and involvement
- 35:20Q&A
related talks
39mDigging Into OpenSearch’s Memory: Lucene and JVM Garbage Collection Under the...
41mScaling Search and Powering Vector-Based Retrieval With OpenSea...
28mRunning OpenSearch at Scale in High-Traffic Gaming Systems
04mOpenSearch Performance and Monitoring
31mMaximize Resource Efficiency with Separated Index and Search Workloads
43mEnhancing Query Performance Through Intelligent Data Co-location
41mSearching large data sets in (near) constant time
37mDerived Source: Slash Storage Costs Without Losing Data in Open...