Integrate Data Quality into your processes
summary
HEDDA.IO supplies a versioned knowledge base of domains, members and rulebooks (regex, lookups, fuzzy algorithms like Levenshtein and phonetics) and exposes Runners to apply those rules where the data lives: a distributed Spark Runner (Databricks/Synapse) for batch scale, a .NET Runner for apps, a Web REST Runner for CI/CD and a single-row processing Azure Function for transactional checks. It returns corrected or invalid row sets, emits profiling statistics back to the portal, but does not do source row-count reconciliation or Power BI semantic-model tests yet.
AI-written summary · about
chapters (29)
- 0:00Introduction and speakers
- 1:03What is data quality
- 3:13Why data quality is overlooked
- 5:06Real-world data quality issues
- 7:18Introducing her IO solution
- 11:10her IO web UI overview
- 13:43Members and synonyms in knowledge bases
- 14:05Rulebooks and versioning
- 15:10Creating business rules
- 16:54Complex rule validation flows
- 18:19Actions and data corrections
- 19:44Dataset-level data quality rules
- 21:09Q&A on data synchronization
- 22:57Integrating data quality into pipelines
- 24:44Introducing Runners for integration
- 26:31Demo in Databricks notebook
- 27:56Performance and scalability
- 31:34Q&A on source system validation
- 32:37Integration tests and system connections
- 34:27Demo in Azure Synapse
- 35:31Demo in .NET Interactive notebook
- 37:19Q&A on state transition constraints
- 38:25Working with corrected data
- 40:51Fuzzy search algorithms
- 43:59Different Runners for use cases
- 45:02Web Runner and REST API
- 45:43Q&A on analytical model testing
- 47:36Statistics and historical tracking
- 49:02Q&A wrap-up and next steps
related talks
50mGet Rid of Your Data Quality Problems
52mData Testing with HEDDA.IO
26mMeeting The Evolving Data Intelligence Expectations
52mElevate your data and AI governance with Databricks Data Intelligence Platform
21mLarge-scale Data Quality Verification (...)
53mData Mesh in Practice - Assuring Data Quality at Scale
44mData Quality Management
22mReliable data engineering made easy