Scaling Shopify's Search: Enhancing Elasticsearch Resilience With Kubernetes and KE...
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 17:45· Level 1 | Hall Entrance S10 | Room C·
summary
Shopify isolates heavy maintenance writes by hosting production and reindex workloads on separate GKE nodepools and autoscaling the reindex pool with KEDA using a Prometheus query (shards ÷ nodes) threshold = 1. Reindex nodes scale from min 3 to max ~200, letting a 100 TB orders reindex run 40% faster while cutting infrastructure cost ~40% and improving production write performance ~65%. The pipeline uses CFKA topics, Elasticsearch allocation rules, and shard relocation to evacuate nodes before scale‑down.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:28Shopify scale and impact
- 2:54Elasticsearch role in search
- 4:00Elasticsearch on Kubernetes
- 5:25SQL to Elasticsearch pipeline
- 7:12Real-time vs reindexing
- 10:41Resilience across regions
- 14:18Search infrastructure scale
- 15:21Shared compute problems
- 17:11Separate real-time and reindex
- 18:37Dedicated node pools with taints
- 20:01KEDA for autoscaling
- 21:50KEDA scaling Elasticsearch
- 24:20Autoscaling workflow
- 26:28Results and improvements
- 27:10Cost savings
- 27:31Q&A
related talks
28mEnhancing Elasticsearch Performance: Innovative Reindexing...
19mBuilding an On-Premise Kubernetes...
33mBuilding Shopify's PaaS on Kubernetes
46mKeep Building Fresh: Shopify's Journey to Kubernetes
41mShopify's Move from the Data Centre to the Cloud
32mAutoscaling Elasticsearch for Logs on Kubernetes
32mAutoscaling Elasticsearch for Logs on Kubernetes
36mAutoscaling Search and Analytics in Kubernetes: The OpenSearch Operator Advantage