A Case Study - Data Movement and Transformation using Azure Data Factory
summary
Case study builds a parameterized Azure pipeline that moves files from Blob Storage into ADLS Gen2 and runs transformations, using Azure Data Factory linked services backed by Key Vault for secrets and Git-based CI/CD. A loading pipeline copies per-business-outcome containers (demo moves 2 files; copy activity auto-creates target container), then a mapping data flow spins up Spark to split a comma-concatenated address into four columns, filter by year/genre, and compute average rating, outputting to ADLS or Azure SQL. mapping data flow Key Vault
AI-written summary · about
chapters (8)
- 0:00Introduction to session
- 1:32Overview of Azure Data Factory
- 12:16Case study setup and design
- 16:22Demo environment setup
- 21:21Loading pipeline walkthrough
- 27:47Transformation with mapping data flow
- 33:46DevOps for Azure Data Factory
- 36:34Q&A and closing remarks
related talks
01h 00mBuilding cloud-native data pipelines using Azure Data Factory
35mData Migration using Azure Data Factory
50mAzure Data Factory patterns and best practices
01h 07mApplication Lifecycle Management in Azure Data Factory
50mDeep {sky}diving into Data Factory in Microsoft Fabric
49mYour first flight with Data Factory in Microsoft Fabric
54mCreating an ADF Pipeline Management System
18mMigrating my favorite past project from On Premise to Azure