DataEngBytes 2023 - MEL-T3-03 - Akira Wang & Abdi Farah
summary
Metadata implementation uses Airflow, Databricks, AWS Glue and system views from Redshift Snowflake and S3 to ingest lineage and job stats into a Postgres central DB and sync with Amundsen, delivering table‑level lineage, Elasticsearch search and a two‑way sync back to the catalog. Adoption rises from ~100 to ~250 users, crowdsourcing hits ~5,000 contributions, estimated savings of 30,000 hours saved for a tooling cost of USD 600/month, and automation reduces manual upkeep.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:04Decentralized data teams challenges
- 2:29Data observability objectives
- 4:37Solution technical overview
- 9:17User personas and needs
- 11:03Metadata modeling with Abdi
- 13:33Metadata ingestion workflow
- 14:36UI and discoverability features
- 16:19Key learnings and engagement
- 21:18Five findings and summary
related talks
24mMetadata driven framework for data discovery and data quality
29mDataEngBytes 2023 - SYD-T1-08 - Rishu Saxena & John Cosgrove
34mTales of Data Architecture Evolution
30mData Engineering building your BI infrastructure from scratch
26mDataEngBytes 2023 - SYD-T3-01 - Scott Eade
14mA Brave New World: How New Lineage Technology Can Reduce Data Spend 40%
17mDemocratising data analytics
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications