Intelligent system optimizations
summary
Shows that tuning ~20 Spark and system config knobs in experiments changes job runtime from about 10 to 90 seconds, and demonstrates TensorFlow/Keras code for both supervised regression (predict runtime from config vectors) and a simple Q‑learning agent. Builds an OpenAI‑Gym style environment that controls instance size/count, balances errors vs cost as reward, and trains an agent to autoscale; naive exploration can crash the system so hierarchical actions and shaping are needed, but learned policies can successfully reduce errors while controlling cost.
AI-written summary · about
chapters (7)
- 0:00Introduction to optimizations
- 0:57Intelligent optimizations with ML
- 3:04Scale and business risk balance
- 5:09Configuring Spark jobs
- 7:59Supervised learning for tuning
- 13:14Reinforcement learning approach
- 20:41Conclusion and next steps
related talks
28mScaling AI Infrastructure
27mLeverage ML to automatically tune K8s apps
28mUnlocking Smarter AI with Inference-Time Scaling
22mAutomating Performance Tuning with Machine Learning
23mContinual Learning for AI Agents: From Failures to Durable Improvements
27mAutomating XGBoost Hyperparameter Tuning with Agentic AI
50mOptimal control, no! Than what? (...)
44mAutomatic parameter tuning for Spark applications