Journey To Cost-Optimized OpenSearch: A Deep Dive Into Segment Replication
summary
Claim: switching from document replication to segment replication plus remote store cuts redundant CPU/IO, raises sustained ingest (benchmarks show ~40% higher indexing, up to ~60% with tuning) and can lower cluster cost substantially. Built features: primary-first shard balancing, translog batching with a translog buffer (default ~600 ms, tunable to ~200–250 ms), backpressure, migration path from document replication, and searchable-snapshot/remote-cluster-state integration. Watch replication lag and remote upload time; use S3/GCS-like blob stores and test SLAs and throughput before migrating.
AI-written summary · about
chapters (13)
- 0:00Cost-optimized OpenSearch intro
- 1:06Document replication model
- 2:31Problems with document replication
- 3:15How segment replication works
- 4:21Benefits of segment replication
- 5:44Hidden benefits of segment replication
- 9:17Trade-offs of segment replication
- 13:16Primary shard balancing
- 14:41Translog buffering latency
- 18:35Safety and monitoring
- 20:42Migration recommendations
- 21:23Complementary features
- 23:50Summary and resources
related talks
31mMaximize Resource Efficiency with Separated Index and Search Workloads
48mMastering OpenSearch Snapshots: From Basics To Advanced Strategies
46mMastering Snapshots in OpenSearch: From Basics To Advanced Strategi...
44mBringing Down the Cost of Your OpenSearch Clusters
31mSharding 2.0: Unlocking Infinite Search Scalability in OpenSearch
43mEnhancing Query Performance Through Intelligent Data Co-location
33mLessons From Migrating To OpenSearch: Shard Design, Log Ingestion, and UI Decisions
37mDerived Source: Slash Storage Costs Without Losing Data in Open...