Is Big Data Performance Reproducible in Modern Cloud Networks?
summary
Study analyzes millions of datapoints from transfers totaling 9 petabytes across three clouds (Amazon EC2, Google Compute Engine, and a Dutch research cloud) and finds two dominant causes of irreproducibility: background stochastic noise that creates ~30% slowdowns and provider QoS like token-bucket throttling that induces up to 10x drops when budgets deplete. Experiments with only 3–15 repetitions often mislead; recommended practices are many more runs, platform microbenchmarks, randomize run order, avoid VM reuse and report confidence intervals.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:48Big data infrastructure in the cloud
- 1:30How do we assess performance in the cloud ?
- 2:20Cloud performance is variable
- 4:09Variability is disconsidered in performance evaluations
- 7:32Experiment Design (1) - Measuring the Cloud
- 11:57Main Findings - Modern Cloud Networks
- 12:27Experiment Design (2) -- Reproducing App Performance
- 12:57How to run repeatable experiments?
- 16:42TL;DR: run more experiments
related talks
16mOptimizing Network Provisioning through Cooperation
30mIdentifying performance bottlenecks in a cloud
26mTuning Your Cloud: Improve Global Network Performance for Applications
31mLive Q+A 3
17mNetHint: White-Box Networking for Multi-Tenant Data Centers
26mTuning Your Cloud: Improve Global Network Performance for Application
20mTurbocharging AI/ML workloads: Revving Up Speed and Resilience
18mPreventing Network Bottlenecks: Accelerating Datacenter Services with Hotspot-Aware...