From Data Centers to Cloud Native - Dave Zolotusky & James Wen, Spotify
summary
Spotify migrates from on-prem data centers to Google Cloud Platform and replaces proprietary pieces with Kubernetes on GKE, gRPC, Istio and managed services. Custom tooling includes open-source Luigi, Heroic, Apollo and a tool called spider that creates ephemeral Hadoop clusters; data migration moves ~20,000 Hadoop jobs while copying data between HDFS and cloud storage. High-throughput services run on Kubernetes (fan-out ~1.5M req/s, mediator proxy ~500k req/s). Migration follows staged rollout: experimental → alpha → beta → golden path with Spinnaker for deployments and plans for schema management and service mesh adoption.
AI-written summary · about
chapters (8)
- 0:00Start
- 14:24Experiment: One Service on Cluster
- 15:34Experiment: Three Services on shared Cluster
- 17:35Alpha: Run Volunteered Services on shared Clusters
- 20:50Alpha: Two High-Traffic Services on shared Clusters
- 23:07Beta Phase Self-Service Migration
- 24:50GA Phase: Golden Path
- 26:57Migration Takeaways
related talks
31mA Journey of Migrating Data Platform from Data Center to Cloud
36mData processing at Spotify - Why portability matters
26mRe-architecting the Data Platform with Spark on Kubernetes
46mContaining an Elephant: How we moved Hadoop/HBase into Kubernetes and Public Cloud
54mData engineering & DataOps: Ask Me Anything
06mYour Backstage, Our Grunt Work: Migrating to Sp...
15mData strategy: Getting ready for Cloud migrations
38mScaling Scala