OPERATIONALIZE THE DATA CATALOG
summary
Argues data catalogs are too reactive and shows a working approach that makes them proactive: combine SQL log parsing with a synchronous Amundsen API hooked into CI/CD so the catalog validates before-and-after SQL changes (prevents dashboard breakages by checking full-change scope). Demonstrates concrete tooling: a dbt macros pattern that reads catalog tags from the warehouse and auto-hashes PII columns, plus a simple GitHub Action + Python check that blocks risky deploys. Claims this reduces backfills and unplanned work and is implementable without inspecting raw data.
AI-written summary · about
chapters (4)
- 0:00Start
- 9:18A Practical Example Proactive Data Uneage
- 12:34What is a Breaking Change?
- 23:41What Makes a Data Pipeline a Good Candidate for Integrating with the
related talks
27mRethinking Data Catalogs: The Promise and Pitfalls
24mMake data central feature
18mData quality: the key to long term happiness
24mDataEngBytes 2023 - MEL-T1-05 - Izzudin Hafiz
33mAccidental DataOps: How we matured our organization with data, and how you can too!
27mLive in the Data Wild West: The Data Contracts Sheriff
16mTurning Data into Action: The Path to Successful Data Governance
54mBehind the Hype - Architecture Trends in Data