Keynote: How Spotify Accidentally Deleted All its Kube Clusters with No User Impact
summary
During a GKE migration Spotify deleted two production clusters by mistake, once via the web console and once when Terraform’s global state file no longer listed the Asia cluster causing Terraform to destroy it. Restore takes include three hours and 15 minutes for the first recovery and an overnight rebuild later, later cut to under an hour after fixes. Root causes are human tab selection, Terraform state file mismatch, missing permissions and resumable scripts; mitigations are backups and disaster recovery tests, declarative cluster code, linters, dry-run outputs, and gradual partial migrations.
AI-written summary · about
chapters (13)
- 0:00Intro
- 1:00Context on Spotify's Compute Environment
- 4:15Cluster Restoration
- 4:54A Month Later
- 7:48Can't Get Any Worse, Right?
- 9:50Developer Impact
- 10:53What We Did Right
- 11:53How Did We Plan for Failure?
- 12:52Partial K8s Migration on Per-Service Level
- 14:37Best Practices
- 15:28Codified Our Infrastructure
- 16:42Performed Disaster Recovery Tests
- 17:31Practice Makes Perfect
related talks
05mHow Terraform Brought Kubernetes Back From Dead
15mKeynote: The Terrifying Migration of 420 Services to Kubernetes
21mHow to Not Destroy Your Production Kubernetes Clusters
35mTo Infinite Scale and Beyond: Operating Kubernetes Past the Steady State
37mBalancing Power and Pain: Moving a Startup From a PaaS to Kubernetes
36m101 Ways to “Break and Recover” Kubernetes Cluster
35mKubernetes journey
38mBuilding and Managing a Centralized Kubeflow Platform at Spotify