Apache Beam pipelines at 100TB+ scale using Apache Spark
summary
Seznam.cz runs Apache Beam on the Spark runner to process 100TB+ daily inputs (pipeline handles 50–100 TB each run, 300M pages/day) and solves severe exponential data skew by combining map-side broadcasts, key-splitting for hot keys, and repartition-and-sort to avoid groupByKey OOMs. Major Spark limits are shuffle-file explosion and per-key memory; a byte-based shuffle (Beam serialization) cuts CPU/GC overhead. Production batch pipelines run daily; profiling with Babar flamegraphs finds fixes like costly combiner serialization.
AI-written summary · about
chapters (16)
- 0:00Intro
- 2:13MapReduce is slow and expensive
- 4:40Python, Java, or Go: It's Your Choice with Apache Beam
- 5:51Turning the internet upside down
- 7:44Exponential data skew
- 7:54joining documents with domains
- 9:12Solution A Map-side Join
- 10:08Solution B Splitting large keys
- 10:45All values for a single key must fit in-memory
- 13:01Memory efficient GroupBy Key
- 14:10Understanding Spark shuffle
- 15:58byte-based shuffle
- 16:31debugging spark pipelines
- 17:23Combine performance
- 17:46heap dumps :
- 18:22Streaming your shared ride
related talks
41mApache Beam: Unified Data Processing
52mApache Beam, a unified model for Big Data processing
21mApache Beam (incubating)
47mApache Beam - what do I gain?
46mBuilding an Apache Beam streaming runtime in GCP
16mYour Data Pipeline with Apache Beam
42mApache Beam: A Unified Model for...
53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]