The Search for Speed
summary
A messy OpenSearch deployment on AWS becomes measurable: added probes, tracing spans and histograms, replaced OpenSearch Benchmark with a custom Go benchmark, and ran hundreds of controlled experiments. Findings: OpenSearch reports per‑shard P99 ≈9–10ms while app P99 ≈1s; outgoing calls hit 200–400ms. Too many shards (latency rises past ~200 shards) and EBS cache misses are the killers. Actions—fewer shards, more RAM‑heavy instances, prune indexes (first ~50% size cut), reshard and shrink nodes—deliver halved latency and roughly 50% cost reduction.
AI-written summary · about
chapters (17)
- 0:00Search speed challenge
- 1:39Figma’s search problems
- 3:47Scientific method for search
- 6:13Accurate search performance data
- 9:20Elasticsearch/OpenSearch metrics
- 12:09Reliable benchmarking tool
- 14:42Testing performance hypotheses
- 15:45Shard count impact on latency
- 17:29Query and config optimization
- 18:34Hardware for search workloads
- 19:58Costs and search reliability
- 20:39Beyond the search backend
- 21:41Index size reduction for speed
- 23:04Faster iteration and experimentation
- 24:28Why generic search advice fails
- 25:52Monitoring pitfalls and SLOs
- 27:41Key recommendations
related talks
44mBringing Down the Cost of Your OpenSearch Clusters
24mFinOps for Observability: Optimizing OpenSearch Costs Without Sacrificing Visibility
38mQuery Insights Deep Dive: From Architecture to Production Optimization
04mOpenSearch Performance and Monitoring
45mMonitoring Your OpenSearch Cluster Effectively
06mWhy OpenSearch?
51mThe Quest for Speed: Journey to 50% Better P99 Times with Go
21mKubernetes Under the Magnifying Glass: Observability With OpenSearch