How Data Hub flies above the clouds!
summary
A centralized Data Hub built in 2020 ingests social media and enterprise sources into Azure Data Lake using the Delta Lake format, processes with Databricks and orchestrates via Data Factory, with metadata in Azure Purview/Unity Catalog, CI/CD in Azure DevOps and infrastructure as code via Pulumi. A standardized Python module enforces naming, ingestion/curation/aggregation stages; a SWAPI demo computes Star Wars BMIs end‑to‑end. Platform runs in production, exposes governed self‑service and cuts operating cost by about 30–40%.
AI-written summary · about
chapters (29)
- 0:00Welcome to Data Hub
- 1:29Presenters and Star Wars theme
- 2:13Data access challenge at Bosch
- 4:00Manual reporting to Data Hub
- 5:46Technical deep dive on architecture
- 6:06Why Azure for Data Hub
- 7:09Core Data Hub architecture
- 9:18Unity Catalog governance
- 11:02Data Hub evolution milestones
- 13:13Delta Lake integration
- 14:40New data source integration
- 16:26Python ETL standardization
- 17:29DevOps for Data Hub
- 19:16Governance demo with cataloging
- 20:20Live demo setup and extraction
- 21:25Azure DevOps branching setup
- 22:51Python data ingestion modules
- 24:14Data transformation and BMI
- 26:22Aggregation optimization
- 28:07Data Factory pipelines
- 29:10Pipeline orchestration and triggers
- 30:39Code review and production deploy
- 31:44Cherry-picking to QA
- 32:25Pipeline execution monitoring
- 33:31Unity Catalog and Power BI access
- 36:46Power BI data visualization
- 37:49Live demo summary
- 38:54Key learnings and challenges
- 39:58Q&A
related talks
40mData Lake House in the Cloud: The Azure Solutions
24mREDLake Azure - Enterprise Data Lake and Business Satellites
43mAnalyzing Big Data in Azure cloud using Spark and Databricks
44mWhat's the Best Big Data Architecture for You?
52mElevate your data and AI governance with Databricks Data Intelligence Platform
50mZero to Lakehouse in Microsoft Fabric
30mData & AI platform components in Azure
36mData Lakes with Azure Databricks