Apache Nemo: A Framework for Building Distributed Dataflow Optimization Policies
Talkat USENIX ATC '19 · on Wednesday, 2019-07-10· at 14:20· USENIX ATC Track II: Grand Ballroom VII–IX·
summary
Apache Nemo is an open-source Java framework (≈32K LOC) that converts jobs from Spark or Beam into an IR DAG and applies compile-time and runtime optimization passes (examples: relay vertex insertion, push-based in-memory shuffle, transient-resource pass) with automated correctness and conflict checks. Evaluations on YARN/Mesos show Nemo with the large-shuffle pass cuts Spark’s disk-seek overhead and reduces job time, Nemo with the transient-resource pass matches a specialized runtime (Paddle) under high eviction rates, and composing both passes further improves performance on combined workloads.
AI-written summary · about
chapters (5)
- 0:00Apache Nemo: A Framework for Building Distributed Dataflow Optimization Policies Youngseok Yang Seoul National University
- 3:00How to optimize distributed execution?
- 5:07Overall Workflow of Apache Nemo
- 7:16(1) Inserts Utility Vertices (2) Annotates Execution Properties
- 12:18Implementation & Evaluation
related talks
44mApache NIFi Deep Dive 300
41mApache Beam: Unified Data Processing
15mNExt generation Meta Operating systems (NEMO) and Data Space: envisioning the future
50mA Java Implementer's Guide to Better Apache Spark Performance
42mStreaming Data Pipelines on Apache Mesos: Lessons Learned
19mDShuffle: DPU-Optimized Shuffle Framework for Large-scale Data Processing
21mApache Beam (incubating)
43mNo shard left behind: Dynamic work rebalancing in Apache Beam