Unity Catalogue and Purview: Data Governance Bedfellows
summary
Live demo integrates Unity Catalog and Microsoft Purview by calling the Unity Catalog REST API and Databricks CLI from Python notebooks, using pyapacheatlas to push custom Apache Atlas type definitions and column-level lineage into Purview. The presenter runs TPCH queries to generate tables (queries take ~40–60s), extracts upstream/downstream relationships and uploads processes so Purview shows mapped source columns. It works end-to-end in the demo, with limits: Purview scan connector needs a self‑hosted runtime, Databricks keeps ~30 days lineage, and scheduled extracts are required.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:49Why data governance matters
- 2:31Unity Catalog and Purview
- 5:43Data lineage explained
- 9:40Extending Purview with custom definitions
- 14:13Demo setup and Python integration
- 18:51Scanning Databricks metadata
- 22:46TPC-H dataset for lineage
- 26:17Mapping source to target tables
- 30:16Column-level lineage in Purview
- 34:10Why lineage automation matters
- 37:01Cost of poor governance
- 39:15Extending beyond Unity Catalog
- 43:35Key takeaways
- 45:04Q&A
related talks
54mData Governance using Azure Purview
05mUnity Catalog Demo of New Features
52mConnect your data artifacts with Azure Synapse Analytics and Azure Purview
15mEvolving Data Governance With Unity Catalog
23mKeynote Analysis
52mElevate your data and AI governance with Databricks Data Intelligence Platform
03mStartup Showcase: Appuri
17mHow will AI transform your data platform?