Data processing at Spotify - Why portability matters
summary
Spotify uses Scio, a Scala API on Apache Beam, to run the same job on Dataflow -> Flink or Spark or Hadoop with no code change. Migration from a 2,500-node Hadoop cluster and ~70 petabytes to cloud takes three years and 80,000 daily copy jobs; today Scio runs ~11,000 pipelines, ~60,000 executions/day and ingests ~600 billion events/day. Running production jobs on Flink succeeds with no code edits but shows 30–60% slower latencies, linear scaling when adding nodes, Kryo serialization conflicts, and extra memory tuning; improving the Beam–Flink runner narrows the gap.
AI-written summary · about
chapters (22)
- 0:00Introduction and overview
- 1:51Use cases for data processing
- 3:56Scale of Spotify's data operations
- 4:38Introducing Scio and Apache Beam
- 6:23Why portability matters
- 13:25Challenges of early choices
- 13:45Spotify's Hadoop cluster in 2009
- 15:30The doomsday clock and Hadoop crisis
- 17:17Migration to the cloud
- 18:20Why migrate from on-premise
- 19:23Gradual migration strategy
- 20:03Lessons learned from migration
- 20:45Principles for future-proofing
- 21:26Experiment with Flink runtime
- 23:14Running jobs on Flink and Dataflow
- 23:57Kubernetes and Flink operators
- 26:04Testing portability at scale
- 26:46Production job experiments
- 31:00Experiment results and insights
- 32:05Why slower can be acceptable
- 34:11Bridging the performance gap
- 35:13Q&A and resources
