Hands-on Workshop: How To Perform Data Quality At Scale
summary
Workshop shows a working pipeline: an open source Soda core Python library translates human-readable agreements into optimized SQL or Spark steps, runs on the data source compute, and pushes results to Soda Cloud; a Docker Soda Agent schedules scans. Over 30 built-in checks cover freshness, missingness, referential integrity and anomaly detection; a demo finds 35% invalid payment methods against a 25% threshold, creates an incident, opens a Jira ticket and resolves it.
AI-written summary · about
chapters (11)
- 0:00Introduction and agenda
- 1:29Three pillars of data quality
- 3:37Soda Core and platform components
- 8:34Live demo setup
- 9:37Finding data in catalog
- 12:49Creating data product agreement
- 17:17Testing and approving checks
- 19:42Data owner approval process
- 20:46Incident detection and resolution
- 24:22Pipeline integration with Soda
- 25:45Q&A session
related talks
21mBuilding Trust in Your Data - Improving Your Data Quality with Automated Framework on AWS
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
38mBI DataOps Development Process
18mData quality: the key to long term happiness
41mData Quality Management: The Good, The Bad, and The Messy
34mData pipelines powered by Open source and fun
44mData Quality Management
15mQuick Guide on Data Quality Assurance