Data Quality Management
summary
Covers a concrete ETL-style pipeline: provenance resource for full traceability, use of the Bulk Data API, and terminology servers to enrich codes and displays. Shows narrative-to-resource extraction (example: IBM Watson Health operation that generates a FHIR bundle), recommends conditional FHIR operations to avoid duplicate creation, and warns that decontainment and deduplication require master indexes. Advises light sanity checks at ingest and full validator runs on export, tracks an error rate over time as the key quality metric.
AI-written summary · about
chapters (25)
- 0:00Intro
- 1:52Scope and Agenda
- 2:20Example #1: Import
- 2:51Example #2: ETL = Extract, Transform, Load
- 3:55Data Quality Management (DQM)
- 8:19Define Quality Rules
- 9:53Tracability
- 11:46On the use of Provenance
- 13:25Cleansing: missing data
- 14:30Cleansing: enriching with master data
- 15:42Cleansing: Enriching using Terminology Servers
- 17:00Cleansing: text v. structured information
- 19:30Cleansing: enriching with references
- 21:21Cleansing: De-Duplication / Reconciliation
- 23:10Cleansing: statistical/Al methods
- 24:48Cleansing: queryability, references
- 26:25Cleansing: De-identification / anonymization
- 27:21Cleansing: removal of unusable data
- 28:43Mapping: Aggregation, Deletion
- 30:42Validation: profile discovery
- 32:05Validation Strategy
- 33:08Validation: flagging the issues
- 34:19Data Quality Metrics
- 35:01Recommendations
- 36:54Summary
related talks
41mData Quality Management: The Good, The Bad, and The Messy
18mData quality: the key to long term happiness
15mQuick Guide on Data Quality Assurance
31mData Reliability Engineering: Secret Of Our Success
01h 00mMDS and DQS - Beyond the TLAs to Data Quality
21mData quality as a product
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
27mThe Sanity Check: Observability for AI Data Pipelines