Open Source for Data Quality
summary
Open source Python workflow that actually names tools and tradeoffs: use pandas-profiling or Apache Superset to profile messy, human‑generated data; define contracts with OpenAPI/Swagger or AsyncAPI and validate every pipeline step with Cerberus or JSON Schema; generate test datasets with Faker or the much faster Mimesis (claim ~10× speed) for load and edge‑case tests. Notes lack a mature open‑source anonymization product, so fake data plus schema validation is the practical route.
AI-written summary · about
chapters (7)
- 0:00Introduction to data quality
- 1:19Qualitative data assessment
- 4:09Data profiling tools
- 6:58Contract-based design
- 9:53Data validation techniques
- 11:53Fake data for testing
- 15:39Conclusion
related talks
29mHow to build a data pipeline without data
01h 29mImproving Open Data Quality using Python
36mData Management with Ruby
05mData Quality and Validation in ML Pipelines
29mSchemas: The Key To Data Happiness
46mBuilding a Self-Managing Test Data Marketplace
42mDeciphering the way data is tested
20mBuilding a Data Pipeline with Testing in Mind