Maximize Resource Efficiency with Separated Index and Search Workloads
summary
OpenSearch 3.0 adds leader‑writer separation that uses remote store and segment replication so indexing and querying run on different node pools. A 250M‑document http_logs benchmark with 4 primary shards and 1 search replica shows converting 2 of 4 data nodes into search nodes cuts query latency and stabilizes CPU without lowering throughput, and dropping data node memory from 32GB to 8GB keeps similar ingest throughput — demonstrating independent scaling, search nodes that can scale to zero, and tangible cost/performance gains.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 2:06OpenSearch leader-writer separation
- 7:25Combined workload challenges
- 9:29Leader-writer separation benefits
- 10:33Terminology and node roles
- 13:23Cluster setup steps
- 14:06Scaling search nodes
- 16:58Search-only mode
- 18:46Case study setup
- 20:57Optimizing separated workloads
- 22:44Hardware optimization
- 23:48Case study results
- 24:10Future evolution
- 25:15Index State Management
- 26:57Summary and call to action
- 27:46Q&A
related talks
41mScaling Search and Powering Vector-Based Retrieval With OpenSea...
31mSharding 2.0: Unlocking Infinite Search Scalability in OpenSearch
44mBringing Down the Cost of Your OpenSearch Clusters
25mJourney To Cost-Optimized OpenSearch: A Deep Dive Into Segment Replication
19mEnhance Search Performance At Scale with
43mMulti-tenancy for all workloads
43mEnhancing Query Performance Through Intelligent Data Co-location
44mScaling OpenSearch at Uber: Operating Resilient, Fault-Tolerant, High-Performance Clus...