Data tiering using data Virtualization in SQL
summary
SQL Server uses PolyBase to query external files and databases without copying data, now shipping a native PolyBase V3 REST architecture in SQL Server 2022 and Azure SQL (preview). Demo reads a public Parquet Seattle dataset (1.8M rows) via OPENROWSET, shows CREATE EXTERNAL TABLE AS SELECT to export archived years into Parquet folders (export produces eight files—one per CPU core), and proves folder partitioning and pushdown cut I/O versus slow UNIONs. Requires DB master key + database-scoped credential (SAS today); Delta write support is roadmap.
AI-written summary · about
chapters (18)
- 0:00Session introduction
- 1:55Data virtualization principles
- 4:30Benefits in SQL
- 6:58PolyBase evolution
- 10:11PolyBase V2 and V3 changes
- 14:47PolyBase V2 vs V3
- 16:14Setting up PolyBase
- 18:40PolyBase prerequisites
- 21:27Data exploration with OpenRowSet
- 23:33Top use cases
- 26:43Demo: Public datasets access
- 30:38Demo: Exporting to storage
- 33:58Query performance comparison
- 36:52Q&A
- 39:18Troubleshooting challenges
- 41:47Learning resources
- 43:13Summary
- 43:54Q&A
related talks
51mData tiering using data Virtualization in SQL
21mThe Power of Polybase with SQL - True data Virtualisation
25mAzure SQL MI & Fabric Fusion: A speed-session
49mAnalytics in SQL Server and Azure SQL Database
46mOpen Table Formats in the Wild: From Parquet to Delta Lake and Back
51mSQL Server 2022 and the Wheel of Power
50mNo no storage limits, just reach for the sky, with data virtualization in SQL MI
01h 02mNot your Father's SQL Server: Developer Innovations in Data Storage