Scaling Scala

on YouTube ↗38m

" summary

Spotify uses Scio on Apache Beam and Google Cloud Dataflow to run thousands of Scala pipelines in production, currently about 11,000 unique pipelines with ~60,000 executions/day processing petabytes and roughly 800 billion events/day. The stack yields strong static typing, macro-derived BigQuery/parquet/Avro mappings and compile-time coder derivation, but faces pain points: closure serialization bugs in distributed runs, cross-publishing for Scala 2.12/2.13, dependency conflicts, and a slow, costly migration path to Scala 3 mitigated by a mono-repo and a custom sbt plugin.

AI-written summary · about

" chapters (15)

1. 0:00Start
2. 4:21Finance Insights
3. 12:09Why Scala
4. 15:20Compiler
5. 17:41Data Processing
6. 22:47Data Engineering Training Internally at Spotify
7. 23:03Train Data Engineers To Do Data Engineering
8. 23:29The Golden Pass
9. 24:06Tech Universities
10. 26:30Civilization Issue
11. 28:01Binary Compats
12. 28:43Migrating to a New Version of Scala
13. 30:19Dependency Conflict
14. 32:44Svg Plugin
15. 36:58Should We Migrate

" related talks

• 03mInterviewJulien Tournay · Scale By The Bay 2018
• 36mData processing at Spotify - Why portability mattersJulien Tournay, Filipe Regadas · Flink Forward Global Virtual 2020
• 24mScio data processing nirvana at SpotifyJulien Tournay, Bram Leenders · Scale By The Bay 2018
• 04mNeville Li InterviewAlexy Khrabrov, Neville Li · Scale By the Bay 2017
• 42mBuilding a Company on ScalaDag Liodden · Scala Days Copenhagen 2017
• 05mInterviewNimbus Goehausen · Scale By The Bay 2018
• 47mScala: The Unpredicted Lingua Franca for Data ScienceDean Wampler, Andy Petrella · Scala Days Berlin 2016
• 37mScala with BazelIttai Zeidman, Ahir Reddy, Oscar Boykin · BazelCon 2017