Azure Data Factory patterns and best practices
summary
Azure Data Factory claims real-world throughput (for example 2 petabyte in 11 days at 2–3 GB/s and a customer doing 1 PB per month increments), exposes DIUs and copy parallelism for tuning, and partitions large tables for parallel extract. Mapping Data Flow generates scripts and runs on scale-out Spark with configurable worker/driver cores and TTL to cut warm startup to ~1–2 minutes. Features include template pipelines for SCD/dedupe, CI/CD per environment, CMK + TLS for metadata, managed identity + Key Vault, and private endpoints via ADF-managed VNet.
AI-written summary · about
chapters (12)
- 0:00Introduction to ADF
- 1:33Core ADF capabilities
- 4:42Data lake migration pattern
- 9:11Hybrid data transfer setup
- 12:21Metadata-driven pipelines
- 15:42Code-free ETL with data flows
- 20:07Data flow configuration demo
- 26:22Code-based transformations
- 28:58CI/CD and security best practices
- 32:02Credential and network security
- 38:17Covid-19 data analysis demo
- 46:15Roadmap and learning resources
related talks
01h 00mBuilding cloud-native data pipelines using Azure Data Factory
40mAzure Data Factory, Azure Databricks, or Azure Synapse Analytics? When to use what.
41mA Case Study - Data Movement and Transformation using Azure Data Factory
50mMicrosoft Fabric Data Factory Q&A
35mData Migration using Azure Data Factory
50mDeep {sky}diving into Data Factory in Microsoft Fabric
49mYour first flight with Data Factory in Microsoft Fabric
20mChoosing between Microsoft Fabric, Azure Synapse Analytics and Azure Data Factory