Enhancing Query Performance Through Intelligent Data Co-location
summary
Claims a segment creation and merge strategy for OpenSearch that co-locates related documents using a user-defined grouping field or script in index mapping, implements a per-group index writer and a context-aware merge policy, and builds an in-memory map at refresh for O(1) segment lookup at query time. Vector search gets separate HNSW graphs per group (less memory and better recall for sparse tenants), log queries scan only relevant segments, benchmarks show ~30k tenants supported per node, ~800 concurrent writers seen, CPU rising modestly and controlled by refresh and aggressive merges.
AI-written summary · about
chapters (7)
- 0:00Context-aware segments intro
- 7:50What are context-aware segments
- 15:22Grouping criteria for segments
- 18:33Indexing with context-aware segments
- 28:04Search with context-aware segments
- 34:09Future plans for segments
- 34:53Q&A on immutability & performance
related talks
41mScaling Search and Powering Vector-Based Retrieval With OpenSea...
31mMaximize Resource Efficiency with Separated Index and Search Workloads
19mScaling Semantic Search to Billions with Neural Sparse ANN
25mJourney To Cost-Optimized OpenSearch: A Deep Dive Into Segment Replication
29mMastering Search Performance: Advanced Query Insights for Production OpenSearch
30mCaching for Speed: Practical Performance Optimization in OpenSearch & Lucene
28mSearch and Sushi; Freshness Counts
39mAccelerating Aggregations with Lucene-Aware Execution in OpenSearch