Containing an Elephant
summary
Runs HBase/HDFS (thousands of machines, multiple petabytes, billions of queries/day) on AWS Kubernetes and proves stateful workloads can work at scale. Uses StatefulSet with a custom custom controller plus on‑delete strategy to enable batched rolling upgrades (hundreds of pods each taking ~5 minutes), adds a per-pod Service to stabilize IPs while fixing DNS caching bugs, uses a sidecar label to avoid repeated leader elections, syncs records via external-dns with validation, and spreads replicas with affinity/anti-affinity and HDFS rack topology; outcome: challenges overcome and production benefits observed.
AI-written summary · about
chapters (7)
- 0:00Managing HBase in cloud
- 1:22Kubernetes for HBase
- 6:36Kubernetes with stateful apps
- 15:19Rolling upgrades with controllers
- 22:25DNS and IP instability
- 28:03Availability across zones
- 31:14Summary
related talks
55mApache HBase: crazy dances on the elephant back
44mSchrodinger's Elephant
24mDynamic Big Data in the Cloud with Kubernetes
34mElephants on Automatic: HA Clustered PostgreSQL with Helm [I]
31mAn Adventure in Multi Cloud with Kubernetes and Distributed Data
37mWhen Distributed Database Meets Cloud - Lessons Learned
27mStateful Workloads at Scale – Extending Kubernetes for Running OLAP Databases as a Service.
49mData Platform Architecture, Evolution, and Philosophy