Constant-Time Aggregations with Star-Tree in OpenSearch
summary
OpenSearch adds a Star-Tree index that materializes multi‑field aggregates inside segments by extending Lucene DocValues to store sums, counts and sketches, shifting cost from document count to field cardinality and delivering sub‑second analytics on billion‑scale datasets. Benchmarks show up to 1000x faster for low‑cardinality dimensions, a configurable maxLeafDocs space/time tradeoff, and a per‑segment upgrade that converts 181M docs (12.7 GB) in about 6 minutes. Caveats: append‑only (no updates/deletes) and high‑cardinality fields blow up storage.
AI-written summary · about
chapters (20)
- 0:00Introduction to constant-time aggregations
- 1:08Problem with linear aggregation latency
- 2:30Limitations of existing solutions
- 3:11How star-tree indexes work
- 5:16Star-tree index structure explained
- 8:55Role of star nodes in queries
- 10:40Configuring max leaf docs parameter
- 12:05Using star-tree in OpenSearch
- 13:10Default vs star-tree query execution
- 15:17Star-tree filter types
- 17:03Handling boolean queries
- 21:18Date histogram queries with star-tree
- 23:28Nested aggregation support
- 25:33Benchmarking star-tree performance
- 28:20Advantages and caveats of star-tree
- 30:28Percentile and cardinality aggregations
- 33:36Retroactive star-tree upgrade
- 35:03Operational lessons and constraints
- 37:10Future roadmap for star-tree
- 38:13Q&A session
related talks
40mLeveraging Lucene Skip Indexes for Faster Aggregations in OpenSearch
30mCaching for Speed: Practical Performance Optimization in OpenSearch & Lucene
37mDerived Source: Slash Storage Costs Without Losing Data in Open...
19mVendor-Agnostic Analytics: Querying Open Columnar Formats With OpenSearch
36mSearch on the fly: how to lighten your Big Data
30mBoosting OpenSearch Performance: Lucene Bulk Collection and gRPC Search in Action
26mOpenSearch and the Path to a Trillion-s...
42mUsing OpenSearch as a Database With DataFusion