Data Testing with HEDDA.IO
summary
HEDDA.IO provides a management UI to define schema domains, compose rule books with conditional rule flows and dataset-level rules, and expose an API and notebook library that runs rules on Spark or a .NET/container runner. Integrations include Azure Data Factory, Databricks, Synapse and Microsoft Fabric; pipelines can trigger validations and stop processing if quality falls below thresholds. It uses fuzzy matching (Levenshtein, phonetics) for synonyms, reports scores (examples shown: 65%, 68%, 88%), can process a million records in under one minute, and stores only metadata back while offering optional record upload. Hosted in the customer Azure subscription with private endpoints; current rule versioning is active/edit with Git planned.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:35Data testing importance
- 8:23Data quality dimensions
- 18:02HEDDA.IO architecture
- 23:21HEDDA.IO walkthrough
- 31:28Automation in validation
- 37:31Notebooks and pipelines integration
- 43:50Fabric and semantic models testing
- 48:26Deployment options & Q&A
related talks
21mData Observability with HEDDA.IO
50mIntegrate Data Quality into your processes
50mNo more data quality surprises in your Databricks, Synapse or Power BI solutions!
52mElevate your data and AI governance with Databricks Data Intelligence Platform
20mDataEngBytes 2023 PER-T1-10 - Matthew Carter
31mBuilding large scale, transactional data lakes with Apache Hudi
24mDataEngBytes 2023 - MEL-T1-05 - Izzudin Hafiz
22mReliable data engineering made easy