Atomic Habits in writing high performing distributed code in the cloud
summary
Pipeline processes 300–600 GB every 30 minutes (peaks at ~100 nodes) and a project table holds 127B rows; switching VM family to T2D machines yields ~45% cost savings. Replacing giant gzipped XML batches with one-XML-per-gzip plus moving CPU-heavy unzip/parse before groupByKey and using combinePerKey with key sharding increases throughput from ~10k → 35k eps (4×) and fully saturates CPUs. Fixing list lookups by using hashset for contains removes a 2× slowdown. JMH benchmarks guide parser and string choices.
AI-written summary · about
chapters (10)
- 0:00Introduction to talk
- 1:58What are atomic habits
- 7:07Systematic approach to performance
- 15:23Optimizing machine types
- 17:15Big data processing analogy
- 21:33User code optimization
- 26:32Batch vs streaming pipelines
- 33:16Group by key performance issue
- 39:17Redistributing keys for speed
- 42:31Key takeaways and Q&A
related talks
56mLarge-Scale Distributed Processing in Google Cloud - 7 lessons...
19mApache Beam pipelines at 100TB+ scale using Apache Spark
01h 06mBest practices when accessing Big Data or any other data!
26mPetabyte Scale Datalake Table Management with Ray, Arrow, Parquet, and S3
23mProfiling & Performance-Tuning Your Hadoop Pipelines
01h 04mBuilding Efficient and Frugal Image Data Processing Pipelines
36mIn-memory data pipeline and warehouse at scale
34mA journey of scaling high-volume low-latency stateful processing