Unleashing the Power of Init Containers: Reducing Database Management Toil at Yelp

on YouTube ↗28m

" summary

Yelp runs init containers to automate and speed Cassandra operations across 70+ clusters and >1,000 nodes, cutting scale-up duplicates by bootstrapping new nodes with CDC disabled during bootstrap so bootstrap streams don’t flood Kafka and scale faster. Upgrades from 3.11 to 4.1 succeed by starting the old version in an init container to preserve IP/version continuity, and SSTable upgrades run controlled per node to avoid disk pressure. Restores use Medusa restores in parallel from S3 and fully automate recovery; the approach works in production.

AI-written summary · about

" chapters (9)

1. 0:00Init containers at Yelp
2. 2:53Yelp’s Cassandra at scale
3. 7:37Kubernetes init containers
4. 10:54Cassandra scaling with init containers
5. 15:17Upgrading Cassandra clusters
6. 21:21SSTable upgrades with init containers
7. 23:06Recovering Cassandra clusters
8. 25:38Best practices for init containers
9. 27:27Q&A

" related talks

• 33mRunning Istio and Kubernetes On-prem at Yahoo ScaleMrunmayi Dhume, Suresh Visvanathan · KubeCon + CloudNativeCon North America 2019
• 47mRunning Databases in Docker at ScaleJoakim Recht · DevOpsCon Berlin 2018
• 32mCassandra at Yammer #bbuzzMichal Rutkowski · Berlin Buzzwords 2015
• 43mLife of a startup...Sjoerd Mulder · Codemotion Amsterdam 2017
• 30mYugabyteDB Elasticity and Resilience in ContainersFranck Pachot · Distributed SQL Summit Asia 2022
• 35mAuto Operation of Hundreds of K8s Clusters in Yahoo!Cheng Chen, Hiroki Katsuta · KubeCon + CloudNativeCon China 2019
• 36m101 Ways to “Break and Recover” Kubernetes ClusterSuresh Visvanathan, Nandhakumar Venkatachalam · KubeCon + CloudNativeCon Europe 2018
• 38mThe Container Operator's ManualAlice Goldfuss · LISA19