Autoscaling Elasticsearch for Logs on Kubernetes
summary
Explains a concrete autoscaling design for Elasticsearch/OpenSearch on Kubernetes: rotate time-based indices by size (recommended 10 GB per shard), update index templates with new shard counts, and use the rollover API plus total shards per node limits to avoid hotspotting. Proposes disk-based scaling, prefers ephemeral local SSDs, and shows a live demo using the Zalando operator (and work toward an Opster operator) that scales a 2-node cluster up to more nodes and back down while updating ISM/index lifecycle settings.
AI-written summary · about
chapters (10)
- 0:00Introduction and presenters
- 1:46Use case for autoscaling
- 3:10Time-based vs size-based indices
- 8:08Auto scaling strategies
- 12:06Best practices for scaling
- 15:20Available Kubernetes operators
- 18:14Live demonstration setup
- 21:36Cluster scaling up demo
- 26:08Cluster scaling down demo
- 29:17Q&A session
related talks
32mAutoscaling Elasticsearch for Logs on Kubernetes
36mAutoscaling Search and Analytics in Kubernetes: The OpenSearch Operator Advantage
29mScaling Search Clusters with Apache Solr and Kubernetes
38mEs-operator: Building an Elasticsearch Operator From the Bottom Up
14mAutoscaled K8s Observability with the Elastic Operator
39mOperating OpenSearch at Scale With Kubernetes Operators: Lessons Learned and Community Insights
32mScaling Shopify's Search: Enhancing Elasticsearch Resilience With Kubernetes and KE...
52mOperating OpenSearch at Scale: Lessons From Managing 10,000+ Clusters Across...