AI-driven query plan optimization in big data processing tools
summary
Claims AI models trained on query telemetry pick Spark configs (e.g. spark.sql.cbo, join reorder, star-schema detection) that cut execution time often 2–3x and — per vendor presentations — up to 25x speedup claim by Databricks. Architecture routes parser/driver metrics into a feature store, trains models (Random Forest / XGBoost / GNN), and returns config recommendations; evaluation uses TPC-DS/TPC-H benchmarks. Notes that open-source projects lack large multi-client telemetry and proposes an external configurator for Spark to collect stats and bootstrap community models.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:57Migration to cloud
- 4:03Classic query optimization
- 6:13Cost-based optimization history
- 9:28Adaptive query optimization
- 10:13AI cost-based optimization
- 13:16Open-source challenges
- 14:44Automated optimization tool
- 16:32AI tool architecture
- 19:28Benchmarking datasets
- 21:44Model selection for prediction
- 23:10Community collaboration
- 25:03Q&A
related talks
40mBest Practices and Optimization with the Spark Engine
18mDatabricks drives predictive maintenance, grid optimization with analytics
24mSmart approach in development and deployment process for ML models
37mOn Building Production-Quality AI Systems
19mDatabricks
15mOptimizing Query Performance in Big Data Systems
36mBig Data Modeling with Spark SQL: Make data valuable
36mDatabricks as the Foundation for GenAI: What’s New and What Works