Large-Scale Distributed Processing in Google Cloud - 7 lessons...
summary
Shows concrete fixes for GCP Apache Beam/Dataflow pipelines: replacing expensive per-record CPU work boosts throughput from ~6k/s to ~47k/s and cuts estimated cost from $0.46 to $0.08; tuning shards (16→32→64) trims runtime; unpacking one large gzip into single-file gzips lifts a job from ~4.2/s to ~24k/s; adding key suffixes resolves hot-key skew and raises throughput 10k→30k while shrinking latency 6min→20–30s; switching CPU family (T2D) yields ~50% perf/$ gain. optimize user code tune shard count avoid hot keys
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker intro
- 2:58Big Data overview and ETL basics
- 5:08Apache Beam framework explained
- 7:39First pipeline example and metrics
- 12:20Optimizing user code for throughput
- 21:07Adjusting shards and file naming
- 24:23Machine type selection for cost
- 28:01XML parsing and compression issues
- 39:11Handling hotkeys in streaming
- 49:11Recap of seven optimization lessons
- 51:48Q&A and bonus tips
related talks
46mBuilding an Apache Beam streaming runtime in GCP
43mNo shard left behind: Dynamic work rebalancing in Apache Beam
53mStreaming data processing pipelines with Apache Beam [in Python, naturally!]
24mApache beam and google cloud dataflow
26mBeam and dataflow: a perfect combination of parallel processing
44mAtomic Habits in writing high performing distributed code in the cloud
19mApache Beam pipelines at 100TB+ scale using Apache Spark
20mStreaming Processing Options in Google Cloud