No stats, no problem: building feedback-driven optimizers for lakehouses
summary
e6data builds a layered optimizer that learns from execution instead of trusting stale stats: an equivalence-set NDV seeding step, a magic‑number sensitivity pass that picks columns for async stats collection, and a LEO learned execution optimizer that canonicalizes queries, stores hashed subtrees with confidence (z‑score), and reuses persisted feedback. Results include 25% faster on TPC‑DS workloads and a real CRM query dropping from 111s→37s, learned stats collected by a background task and applied at plan time.
AI-written summary · about
chapters (15)
- 0:00Introduction to e6data
- 2:02Optimizer basics and stats
- 4:51The lakehouse challenge
- 6:58A layered survival model
- 11:13Equivalence set inference
- 12:50Magic number sensitivity
- 13:48Sensitivity analysis examples
- 17:15Learned execution feedback
- 19:00LEO workflow and architecture
- 20:48Canonicalization strategies
- 23:59Context blending
- 26:09Confidence scoring
- 27:24Performance results
- 29:18Future research and conclusion
- 30:43Q&A session
related talks
01h 04mQuery Optimization and Acceleration at Dremio
41mHydra's Heads – Vladimir Ozerov
32mOpenTelemetry, databases, and machine learning
01h 00m7 Tips to Performance Tuning, Optimization, and Everything
53mThe Most Important Performance Factor in SQL Server and Azure SQL
31mFrom Data to Precision: Fine-Tuning LLMs for Proprietary Query Languages
14mDeductive Optimization of Relational Data Storage
28mMysql Optimizer Features in 5.7 and 8.0