No more data quality surprises in your Databricks, Synapse or Power BI solutions!
summary
Uses Great Expectations inside Databricks/Spark to run column and table checks on a 10 million‑row sample, generate and save JSON expectation suites, and build browsable data docs (HTML) immediately. Demonstrates the onboarding data assistant that bootstraps dozens of rules per table (13 source tables profiled in a day), shows layered rules with critical vs warning checks and thresholds (example: 0.95), and wires Python calls to the Power BI REST API for model queries (requires service principal, limits 120 req/min, one table per call).
AI-written summary · about
chapters (9)
- 0:00Introduction to data quality
- 2:00Understanding data validation
- 7:05Defining data expectations
- 12:10Practical implementation
- 20:44Managing the data context
- 30:28Automating expectations
- 36:29Handling validation feedback
- 41:03Pipeline integration
- 44:41Validating Power BI models
related talks
15mQuick Guide on Data Quality Assurance
17mHow will AI transform your data platform?
42mDeciphering the way data is tested
52mElevate your data and AI governance with Databricks Data Intelligence Platform
22mReliable data engineering made easy
45mFrom hand-down maps to AI/BI: a journey toward Databricks
18mDatabricks drives predictive maintenance, grid optimization with analytics
45mDo those numbers look right? How to ensure quality, and avoid inaccuracies in your data insights.