Test-Driven Data Analysis in Python
summary
Claims a concrete methodology plus an open-source Python toolset: the tdda library for discovering constraints from training data, Rexy for inferring regular expressions from examples, and GenTest for automatic test generation. Demo shows discovery producing 78 constraints that pass on the 92-element CSV, then 17 constraint failures when verifying a 118-element file, and iterative editing to relax too‑tight rules. GenTest writes runnable tests but sometimes misclassifies random outputs, so expect useful automation plus manual tuning.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 1:34Test driven development
- 4:25From test driven development to test driven data analysis
- 6:11Concept behind TDDA
- 7:34Test driven data analysis methodology and library
- 12:28Constraints for data checks
- 15:18Garbage in garbage out
- 17:04Constraint types for tabular data
- 19:52Discovering constraints with TDDA
- 25:16Refining constraints
- 34:11CSV and Parquet files
- 35:16Commercial data verification
- 36:19Sherlock Holmes on constraints
- 38:24Rexie for regex generation
- 42:51Batch verification
- 44:02Comparison with Great Expectations
- 45:47Testing data pipelines
- 47:34Reference testing
- 51:05Reference test support
- 1:00:49Checking data frames
- 1:06:58Automatic test generation
- 1:14:20Type six errors
- 1:25:18Q&A
