Data tiering using data Virtualization in SQL
summary
SQL Server shows a working data-virtualization stack: PolyBase V3 runs natively in the engine with a REST/S3 object-storage connector, supports Parquet/CSV reads and CTAS export to Parquet, and coexists with the older V2 ODBC service. A demo queries a public Seattle safety Parquet set (1.8M rows, ~14s), exports by year into partitioned folders (creates eight files — one per CPU) and proves file-path partitioning vastly reduces round trips. Delta writes are not yet supported.
AI-written summary · about
chapters (18)
- 0:00Session introduction
- 1:55Data virtualization principles
- 4:30Benefits in SQL
- 6:58PolyBase evolution
- 10:11PolyBase V2 and V3 changes
- 14:47PolyBase V2 vs V3
- 16:14Setting up PolyBase
- 18:40PolyBase prerequisites
- 21:27Data exploration with OpenRowSet
- 23:33Top use cases
- 26:43Demo: Public datasets access
- 30:38Demo: Exporting to storage
- 33:58Query performance comparison
- 36:52Q&A
- 39:18Troubleshooting challenges
- 41:47Learning resources
- 43:13Summary
- 43:54Q&A
related talks
51mData tiering using data Virtualization in SQL
50mSQL 2022 and S3 – Dell’s overview of consuming object data via Microsoft SQL
21mThe Power of Polybase with SQL - True data Virtualisation- 21mThe Power of Polybase with SQL - True data Virtualisation
32mConnecting Silos In Real-time With Data Virtualization
49mAnalytics in SQL Server and Azure SQL Database
46mOpen Table Formats in the Wild: From Parquet to Delta Lake and Back
22mData virtualization - Unlocking data - From silos to success?