Es-operator: Building an Elasticsearch Operator From the Bottom Up
summary
Zalando builds a Kubernetes-native Elasticsearch operator that manages 100s-of-node clusters (peaks to ~512 on Black Friday), wraps StatefulSets with a custom CRD ElasticsearchDataSet, and performs safe drains by scaling out, calling the Elasticsearch API to exclude nodes, waiting for shards to move, then deleting empty pods. It implements two-dimensional autoscaling (nodes and index replicas), persists progress in the CRs so the operator can die-and-recover, and is running in production saving substantial cost.
AI-written summary · about
chapters (26)
- 0:00Introduction
- 1:07Who am I
- 1:59What is Zalando
- 2:19Currently serving 26 million customers
- 2:45Kubernetes
- 3:52Search
- 4:38Black Friday
- 5:01Instances
- 5:48Goals
- 7:04Stateful Sets
- 10:42Operator
- 12:15BeZero
- 13:29Stateful
- 13:58Elastic Search Data Set
- 14:57Elastic Search Data Sets
- 17:15Scaling
- 19:47Scaling down
- 21:40Scaling replicas
- 22:24Scaling in production
- 23:49Lessons learned
- 26:44Other projects
- 28:04Questions
- 32:10Elasticsearchs own operator
- 33:20How many clusters are managed
- 34:25Scalability issues
- 35:20CoopDB operator
related talks
32mAutoscaling Elasticsearch for Logs on Kubernetes
40mOperating Opensearch - The Kubernetes Way
35mOrchestrating Resilient Data: Harnessing the Strength of Kubernetes with Operators
38mScaling an online search engine to thousands of physical stores
50mThe hell of SQL DBs on Kubernetes versus the promise of operators
44mDatabases on Kubernetes: Why You Should Care!
25mLessons Learned from Building a Database Operator
27mDeploying Databases on Kubernetes: Helm vs. Operators - Eric Schäffer and Manuel Swiercz