Big Data Pipelines With Scala - Part 2 of 2
summary
Spotify processes many terabytes a day and uses Scio Scala API for Apache Beam to build pipelines that run on the Google Cloud Dataflow runner. The repo shows concrete techniques that work: hot‑key fan‑out, hash/broadcast joins and a count‑min sketch hybrid to avoid shuffle blowups, windowing with watermarks and triggers, unit and property tests plus crash tests, custom Kryo encoders for 20% speedups, and accumulators/metrics. Examples run on Dataflow and report real operational tradeoffs and measured gains.
AI-written summary · about
chapters (6)
- 0:00Start
- 3:15Hash Join
- 6:12Partial Joint
- 11:46Semigroups
- 23:37Speculative Execution
- 28:10How To Choose the Time Stamp
related talks
36mData pipelines engineering made simple with Scala
01h 01mCurrent Methods for Building Data Pipelines
52mThe Evolution of Hadoop at Spotify Through Failures & Pain
29mData Acquisition: Overcoming Challenges
51mStream all the things! - life at the end of the firehose
42mBuilding a Streaming (E-health) Data Pipeline: When and How?
01h 01mStream Data Processing for Fun and Profit
19mDemystifying Modern Data Pipeline Architecture