Whiz: Data-Driven Analytics Execution
summary
A prototype called Whiz cleanly separates computation from intermediate data and adds a data service that collects runtime stats, applies execution and modification predicates, and performs late binding of task parallelism and placement. Built by modifying Tez/YARN and evaluated on a 50-node cluster with TPC-DS queries, PageRank, and streaming workloads, it reports 1.3–2x faster job completion and ~1.4x better cluster efficiency by reducing skew, improving locality, and avoiding hotspots.
AI-written summary · about
chapters (16)
- 0:00Intro
- 2:26Data Opacity + Compute Rigidity
- 2:41Static Execution Structure
- 3:00Compute-driven Scheduling
- 3:29Analytics Limitations: Root Causes
- 4:13Whiz Approach
- 4:31Intermediate Data Visibility
- 5:14Runtime Physical Graph Generation
- 5:45Data-driven Computation
- 6:12Whiz Job Execution Pipeline
- 7:30Whiz Data Service How to organize the intermediate data from a job stage ?
- 8:13Whiz Execution Service
- 8:56Whiz Evaluation: Implementation and Setup Implementation: Modified Tez and YARN
- 9:11Whiz Evaluation: Batch Analytics and Graph Analytics
- 9:48Whiz Evaluation: Sources of Improvement
- 10:49Summary Compute-centric execution engines hurt flexibility, performance and efficiency - Tight coupling between compute and intermediate data - Intermediate data agnosticity
related talks
20mJourney of Building a Modern Data Prep Tool...
42mComposable Parallel Processing in Apache Spark and Weld
27mHarnessing the Power of Dual Query Engines.
36mIn-memory data pipeline and warehouse at scale
42mWhy Choose One: Multi-Engine Analytics with Apache Wayang
43mData warehousing reinvented for today's needs
29mApache Spark Performance Tuning Session with Delight
34mHive on Spark #bbuzz