Automatic parameter tuning for Spark applications
summary
Describes an automated system that chooses optimal Hadoop/Spark resource configurations for hundreds of daily and thousands of hourly Spark jobs with different SLAs. Uses automatic tuning to assign resources per model/job in a Hadoop cluster so engineers don’t have to manually tune each Spark application.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 7:18Manual tuning problem
- 10:13Spark memory optimization
- 17:18Spark logs metrics analysis
- 22:14Automatic tuning system architecture
- 27:10Implementation challenges and solutions
- 32:53Results and conclusions
- 37:09Q&A
related talks
43mOptimizing Spark applications from simple to complex with examples
17mSelfTune: Tuning Cluster Managers
27mDr. Elephant - Self-Serve Performance Tuning for Hadoop and Spark
28mAI-driven query plan optimization in big data processing tools
30mOptimize Spark Jobs Using Its UI
24mIntelligent system optimizations
41mAutomated Performance Tuning with Bayesian Optimization
29mContinuous tuning of Microservices via Bayesian Optimization