Analytics at the speed of Direct Lake
summary
Direct Lake uses Parquet files plus V order compression to page columns into memory on demand, so models start empty and only queried columns load (column-specific paging). Example: CSV -> Parquet shrinks 880 GB to 268 GB, V order to 84 GB. It supports fast import-like queries but forbids calculated columns, composite models and some datatypes, and enforces guardrail fallbacks (eg. very large tables or too many files fall back to DirectQuery — F64 limits around 1.5B rows). Demos use Fabric Lakehouse/warehouse, notebooks and Python (simpai) DMVs.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:15What is Direct Lake
- 2:19Direct Lake vs import and DirectQuery
- 4:50Direct Lake benefits and limits
- 8:46Why Parquet and V Order
- 11:36Parquet structure and encoding
- 16:36Cardinality impact on performance
- 18:40What is V Order
- 20:25Direct Lake model best practices
- 23:13Demo setup and difficulties
- 26:31Creating Direct Lake model
- 32:58Hydrating columns in memory
- 35:03Query execution and paging
- 37:29Memory management and eviction
- 38:54Framing for data consistency
- 40:59How framing works
- 43:11Manual and automatic framing
- 44:34Guard rails and fallback
- 46:40Summary and Q&A
related talks
46mMicrosoft Fabric Synapse data warehouse Q&A
47mData analytics for the era of AI
39mDelta Lake: Diving into Data Lakes Without the Downsides
46mOpen Table Formats in the Wild: From Parquet to Delta Lake and Back
24mData Mesh and Lakehouse
30mEmpower every BI professional to do more with Fabric
22mData Exploration Made Easy with Open Source
28mLakehouse Architecture with Delta Lake and Databricks