Characterizing the performance of Big Data processing platforms
summary
Compares Hadoop MapReduce terasort, Hive TPC-DS and Databricks Spark benchmarks on AWS and reports that CPU and memory scale roughly linearly but network shuffle dominates once instances hit practical throughput limits (200–500 Mbps on d2.xlarge), creating superlinear job times; uses cloudwatch/ganglia and Hadoop YARN/timeline REST metrics and a continuous intelligence dashboard to correlate resource patterns, shows greedy memory consumption, and argues dynamic tuning rather than static configs, with a working prototype for prediction.
AI-written summary · about
chapters (9)
- 0:00Performance characterization
- 2:56Methodology and benchmarks
- 5:46Analyzing resource bottlenecks
- 8:46Continuous intelligence
- 11:55Resource consumption patterns
- 15:02Infrastructure investment
- 21:20Query-based system testing
- 26:38Integrated data analysis
- 31:22Future modeling and outlook
related talks
40mPerformance Testing and Big Data
30mThe Future of Scala Performance on Big Data
27mTales at Scale: Analytics at 1000 QPS and Beyond
40mBest Practices and Optimization with the Spark Engine
48mContinuous Optimization for distributed BigData analysis
38mApache Druid: Why Speed In Analytics Matters
25mBest practices for advertising and marketing technology on AWS
01h 06mBest practices when accessing Big Data or any other data!