Kubernetes: Trials and triumphs

on YouTube ↗36m

" summary

Three years of running CockroachDB on Kubernetes produces concrete outcomes: an in-house K8s operator (shipped end of 2021) that automates node decommissioning/recommissioning via Cockroach gRPC, multi-region multi-cloud deployments using Istio-based service discovery, and a smooth cloud migration with canary traffic. Disk resizing now uses a PVC autoscaler to increase sizes in minutes and shrink by copying (7 minutes–4 hours per node), vertical CPU autoscaling based on 7‑day p50/p90 reduces waste, and spot nodes save cost in staging but remain risky for production.

AI-written summary · about

" chapters (20)

1. 0:00What is City Storage Systems and how do they use CockroachDB?
2. 1:08Why are they running CockroachDB?
3. 2:19The benefits of Kubernetes
4. 3:12Provisioning a CockroachDB cluster on Kubernetes
5. 4:16Day 2 Operations - Scaling disks
6. 6:35Day 2 Operations - Scale in horizontally
7. 7:53Day 2 Operations - Scale out horizontally
8. 9:18Lessons from the journey so far
9. 10:44Kubernetes operator pattern
10. 11:39Writing a CockroachDB operator
11. 13:13Scaling all the things - Adding and removing nodes
12. 15:22Scaling all the things - Disks Take 2
13. 16:09Lessons learned from scaling
14. 17:19Multi-region deployments and disaster recovery
15. 19:28Cloud migration with CockroachDB and the Kubernetes Operator
16. 21:47Lessons learned from building with a cloud-native mindset
17. 23:40Vertical auto scaling - CPU recommendations
18. 25:48Scaling Disks - Take 3 - Scale up
19. 28:34CockroachDB on Spotnodes
20. 30:59Key takeaways

" related talks

• 33mWhat We’ve Learned Building a Multi-Region DBaaS on KubernetesJosh Imhoff · Virtual Rejekts 2020
• 31mFuture-proofing operational data at scaleSpencer Kimball · RoachFest 2025 Las Vegas
• 29mHow City Storage Systems designed its Data Infrastructure to minimize toilAlex Filipchik · RoachFest 2024: Menlo Park
• 26mChallenges of Managing Stateful Applications in KubernetesTamal Saha · KCD Dhaka 2024
• 22mRunning distributed stateful system on Kubernetes - Beyond StatefulsetsBabu Srithar · Kubernetes Community Days Bengaluru 2023
• 36mKubernetes Runs Anywhere, but Does your Data?Jared Watts · KubeCon + CloudNativeCon Europe 2018
• 48mBuilding mission-critical applications with CockroachDBAndrew Baptist · AWS re:Invent 2023
• 45mLearning How to Pronounce Kubernetes to Production in 3 Months!Sheriff Mohamed · KubeCon + CloudNativeCon North America 2016