Metadata driven framework for data discovery and data quality
summary
A metadata-driven system harvests Redshift, Airflow and job stats with Python/Spark into a Postgres central store, surfaces results in Amundsen (Elasticsearch search) and feeds Anomalo for data-quality checks, producing a metadata collection DAG and a slack analysis DAG that links questions to tables. Outcomes: lineage for 93 engineering tables, over 1,400 metadata fields collected, programmatic API wiring to create checks, and a weighted scoring model that quantifies metadata and quality completeness.
AI-written summary · about
chapters (6)
- 0:00Start
- 1:02Data Empowerment Outcome focused Data Management
- 5:11Slack Analysis DAG (Conceptual)
- 10:02Data Quality is Everyone's Responsibility
- 15:48Metadata scores dashboard
- 17:33Future Roadmap
related talks
24mDataEngBytes 2023 - MEL-T3-03 - Akira Wang & Abdi Farah
30mData Discovery with Amundsen
46mJourney of metadata
40mData Intelligence Platform on AWS
37mMetadata Automation in the Data Mesh
29mMetadata Systems for End-to-End Data & Machine Learning
33mDiscover How to Create Your Own Metadata-Driven ML Platform from Scratch
53mData Mesh in Practice - Assuring Data Quality at Scale