Profiling & Performance-Tuning Your Hadoop Pipelines
summary
Etsy runs hundreds of Cascading flows (thousands of Hadoop jobs daily) and finds concrete wins: replacing Scala parser combinators with a left-to-right parser yields 10x faster initial log parsing, swapping generic MapWritable for a bespoke string→string writable removes costly reflection, and wrapping Java event objects for Scala avoids full-map copies. Profiling uses YourKit with hundreds of thousands of JVM snapshots dumped to HDFS and re-aggregated by Hadoop jobs to pinpoint hotspots.
AI-written summary · about
chapters (7)
- 0:00Introduction and context
- 1:29EMR to in-house cluster shift
- 3:55Log parsing performance issues
- 8:12Writable map serialization flaws
- 12:10Java and Scala interoperability costs
- 15:28Profiling large-scale Hadoop jobs
- 19:05Q&A on serialization and Avro
related talks
44mAtomic Habits in writing high performing distributed code in the cloud
21mAvoiding Spark Pitfalls at Scale
50mOur experience at Etsy com with building “Big Data” analytic capabilitie
24mTaming Data Skew in Production ML Pipelines
50mPerformance: Avoiding Pitfalls, Diagnosing Issues & Scaling for Growth
36mIn-memory data pipeline and warehouse at scale
20mProfiling and optimizing a Spark job with Babar #bbuzz
23mThe Future of Scala Performance on Big Data