Query Optimization and Acceleration at Dremio
summary
Dremio describes a SQL lakehouse engine built on Apache Calcite and Apache Arrow with Gandiva LLVM codegen, Parquet storage and Apache Iceberg table metadata. It reports up to 90x speedups from codegen on large expressions and an 8x win from runtime filtering (often much larger in bad cases). Stats use HyperLogLog, t‑digest and items sketches; reflections (materializations) plus parallel Iceberg manifest scans, NVMe caching with consistent hashing and 1 MB chunks reduce I/O and planning bottlenecks.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 3:56Open Source Technologies
- 5:50Who are we
- 8:02Query planning
- 10:09Heuristic vs Volcano planner
- 13:38Statistics
- 22:45Analytics
- 25:21Examples
- 30:49Challenges
- 38:22On Time Filtering
- 39:32Runtime Filtering
- 40:40Complex Expressions
- 45:06Columnar
- 46:39Cellularium
- 52:33Cache Optimization
- 56:34Chunk Size
related talks
13mBuild your open data lakehouse with Dremio and AWS (PRT085)
45mBuilding an Open Data Lakehouse using Apache Spark, Apache Iceberg and Dremio- 45mBuilding an Open Data Lakehouse using Apache Spark, Apache Iceberg and Dremio
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics
22mAccelerating Analytics with Apache Arrow
41mNo stats, no problem: building feedback-driven optimizers for lakehouses
37mApache Iceberg: An Architectural Look Under the Covers
34mAnalytics and AI with PostgreSQL - Mission Possible!