Keep the Data Flowing
summary
Workshop demonstrates deploying Great Expectations to automate data checks: init a data context, run the profiler, generate data docs and an expectation suite in source control. Concrete fixes include catching zero longitudes, inconsistent timestamp formats, and Celsius vs Fahrenheit swaps; a custom expect_column_pair_to_be_unique expectation checks id → (lat,long) uniqueness. Team reports over 40 experiments feeding this effort, and validation runs produce HTML reports and pass/fail JSON that gate pipelines so bad batches stop and good ones proceed.
AI-written summary · about
chapters (9)
- 0:00Welcome to Pi Data Festival
- 1:53Data validation with Great Expectations
- 7:11Data pipelines need validation
- 23:57Workshop environment setup
- 25:47Basic validation workflow
- 51:57Production-ready validation suites
- 1:12:54Challenge: Finding weather data mistakes
- 1:24:42Summary and takeaways
- 1:32:09Q&A
related talks
42mDeciphering the way data is tested
37mEngineering data quality
50mData-Driven Engineering
34mWhat are data unit tests and why we need them
35mBad data - anecdotes and examples from the real world
20mBuilding a Data Pipeline with Testing in Mind
30mFailing Fast with Data: Bringing CI to the Land of Data
25mWhy Software Engineering Best Practices Fail in Data Engineering