DataDevOps for the Modern Data Warehouse on Microsoft Azure
summary
Modern Data Warehouse on Azure implements an end-to-end CI/CD pipeline that actually works: REST ingestion from the Melbourne parking sensors API lands into bronze/silver/gold zones on Data Lake Gen2, Databricks notebooks call a packaged Python wheel so transformations are unit-testable, and PolyBase loads into SQL Data Warehouse with ARM-deployed Data Factory and Azure DevOps releases. Key Vault stores secrets, App Insights emits row-count metrics, and the demo shows a full staged deploy to staging and monitoring—an approach that yields replayable pipelines and automated ops via Data Factory + Databricks + Azure DevOps.
AI-written summary · about
chapters (32)
- 0:00Introduction
- 0:35Agenda
- 2:57Traditional Data Warehouse
- 3:33Data Lake
- 6:12Data Lake tiers
- 7:52Validate
- 9:27Ensure
- 10:19Demo
- 17:34Common Question
- 17:49DevOps
- 18:21Data Pipelines
- 20:10Data Bricks
- 24:08Sequel Data Warehouse
- 27:03Data Factory
- 27:40Data Factory Workflow
- 30:55Key Vault
- 33:29Sample Release
- 37:11Commit to Master
- 40:06Static artifacts
- 40:37Seco packages
- 41:33Creating a release
- 42:56Release Pipeline
- 43:34Staging Environment
- 44:36Variables
- 45:15Release
- 46:21Override parameters
- 47:33PowerShell script
- 48:07Backpack task
- 48:54Monitoring
- 49:44Monitoring Metrics
- 50:46Application Insights
- 52:55Summary
related talks
50mBest practices for implementing a modern data architecture with Azure
01h 05mWorkflows of Highly Functional App & Data Engineering Teams
50mA Deep Dive into DevOps Practices with Azure SQL
46mAzure Key Vault, Azure Dev Ops and Data Factory
51mFrom DevOps for data to DataOps
48mConnecting the dots of Azure SQL DevOps
28mDevOps in a Data Warehouse: Inside Out
19mDemystifying Modern Data Pipeline Architecture