Deep {sky}diving into Data Factory in Microsoft Fabric
summary
Demo moves a 1 TB CSV from Azure Blob to Fabric Lakehouse as a binary copy in under 4 minutes using auto ITO and 128 threads, reporting 1 TB binary copy <4 minutes and a breakdown showing read/write overlap. Other examples: parquet→lakehouse ~5 GB/s (~3.3 min), CSV→table 1.5B rows in ~13 min (~1.2 GB/s), Azure SQL DB 24 GB in <2 min. Metadata-driven pipeline uses a control table + lookup + for-each to copy many SQL tables to Lakehouse as CSVs and succeeds. Dataflow Gen2 shows staging to lakehouse = 2s for group-bys, and partitioning into 10 buckets yields ~7x speedup; UI also extracts web tables via column-by-example.
AI-written summary · about
chapters (11)
- 0:00Microsoft Fabric intro
- 1:10Data Pipelines deep dive
- 7:53Copy performance benchmarks
- 11:45Metadata-driven pipelines
- 20:01Dataflow Gen2 optimization
- 27:28Staging for faster transforms
- 32:07Partitioning & parallel processing
- 34:34Lazy eval & query optimization
- 35:37AI in Power Query
- 40:16AI web data extraction demo
- 45:21Q&A
related talks
27mModernize your data integration to enable petabyte scale analytics with Microsoft Fabric
47mData analytics for the era of AI
30mDataEngBytes 2023 - BRI-T1-04 - Mimoune Djouallah
19mAccelerate your data potential with Fabric
50mAzure Data Factory patterns and best practices
50mMicrosoft Fabric Data Factory Q&A
46mMicrosoft Fabric Synapse data warehouse Q&A
40mData Lake House in the Cloud: The Azure Solutions