Building an Azure Data Analytics Platform End-to-End
summary
Builds a working end‑to‑end Azure analytics platform using Azure Data Factory with self‑hosted IR (up to four nodes), private networking/ExpressRoute, IoT Hub/Event Hub + Stream Analytics, Synapse or Databricks for Spark compute, and ADLS Gen2 with Delta Lake storage. Shows versioned solutions v1→v6, a privacy‑store pattern that keeps PII separate and returns anonymized values, and operational concerns: 40 activities per pipeline limit and painful defaults like Databricks vnets reserving ~32,000 IPs. Demonstrations are practical and repeatable.
AI-written summary · about
chapters (22)
- 0:00Intro
- 4:42Question: What is our goal?
- 6:33Paul's Reference Architecture
- 8:09What is your primary design focus?
- 11:10Data Extraction & Ingestion-Spec v1
- 12:31Data Extraction & Ingestion - Spec v2
- 12:46Data Extraction & Ingestion - Solution 2
- 13:35Data Extraction & Ingestion - Spec v3
- 13:58Data Extraction & Ingestion - Solution 3
- 15:08Data Extraction & Ingestion - Spec v4
- 15:23Data Extraction & Ingestion - Solution 4
- 16:37Data Extraction & Ingestion - Spec v5
- 17:15Data Extraction & Ingestion - Solution 5
- 18:05Data Extraction & Ingestion - Spec v6
- 18:26Data Extraction & Ingestion - Solution 6
- 21:01Scaling Up and/or Scaling Out
- 21:54What Compute Type of Compute?
- 22:39Data Transformation - Compute
- 25:03Data Transformation - Storage & Format
- 30:32Loading & Consuming Data
- 34:56Consuming Our Lake House in Azure
- 37:43Overall Architecture
related talks
47mImplement a Serverless Data Analytics strategy with Microsoft Azure
49mBuilding Advance Analytics pipelines with Azure Databricks
57mOptimize “All Data” with a Modern Data Warehouse Solution
55mAzure SQL database, Azure Synapse Analytics, Azure Databricks?
28mArchitecture patterns of Azure Cosmos DB & Azure Synapse Analytics
19mDemystifying Modern Data Pipeline Architecture
37mBuild a Modern Data Platform on AWS
07mOptimizing Data Pipelines with Azure Data Factory