Better, faster, stronger Coder. Improving data processing @Spotify
summary
Claims concrete gains: Scio 0.7.0 adds compile-time coder derivation via Magnolia and Scala macros, falling back to Kryo only for unsupported Java types. Benchmarks show Magnolia-derived coders encode faster and produce smaller bytes than Kryo, and a real pipeline drops from ~60 to ~45 minutes while worker count falls from 200 to 119 (~25% speedup and resource reduction). The build fixes serialization unpredictability, reduces binary size, and surfaces compile-time diagnostics so problematic classes are caught before runtime.
AI-written summary · about
chapters (9)
- 0:00Start
- 1:01about:ito
- 2:28about:scio
- 11:15Serialization is HARD
- 12:30Spark support for Scala 2.12
- 16:34Yay for Kryo?
- 22:02Magnolia
- 24:28Performances
- 27:35Real pipeline
related talks
01h 03mBig Data Pipelines With Scala - Part 1 of 2
52mPractical Binary with Scodec
59mSpark 2.0
28m10 years of Parquet- what's next?
50mBetter Spotify Playlists through Music Theory & Discrete Optimization
42mScala: The Unpredicted Lingua Franca for Data Science
01h 02mLightning Talks (02) & Closing Notes
15mApache Beam and Google DataFlow IRL with Scala Scio