Context-Aware Segments: Solving the "Scatter-Read" Problem
summary
Introduces Context-Aware Segments (OpenSearch RFC #18576, Lucene Issue #13387) that enforce document locality by routing documents to group-specific writers via a user-defined grouping criteria and exposing those segments through a composite index writer; segment-level metadata enables the query coordinator to perform pre-search pruning and skip HNSW vector blobs per segment. Implementation yields measured wins: range queries scan ~3x fewer docs, sorts ~15x, aggregations ~40x, compression improves ~20%, and end-to-end queries see ~30–40% latency improvements.
AI-written summary · about
chapters (14)
- 0:00Introduction and speakers
- 1:12How OpenSearch organizes data
- 3:16Problem with scatter-gather pattern
- 5:43Use case: HTTP log analytics
- 7:10Use case: Multi-tenant vector search
- 9:18Introducing context-aware segments
- 16:01Grouping criteria explained
- 18:06Architecture of context-aware segments
- 21:18Indexing with grouping criteria
- 24:48Refresh and double buffer pattern
- 30:06Handling document updates
- 36:03Compaction and merge policies
- 38:12Search optimization with grouping
- 40:18Performance improvements and conclusion
related talks
31mNext-Gen Search: How Uber and the OpenSearch Community Built a Cloud Native 3.0
24mReimagining Search : Scaling OpenSearch Across an Enterprise - Platform C...
57mOptimizing Frontend Performance of a High-Load Search
39mOptimizing CephFS with Combining MDS QoS Scheduling and Static-Dynamic Subtree Partitioning
39mRelevant search results (with elasticsearch)
01h 01mBounded Contexts: Manage the Understandability of Your Systems
44mBuilding a Modern Search Engine: The Road Ahead
44mFuzzy Search in the Browser Over Half a Million Lines: Building an Index on a CDN