How to Test BigData/ETL Solutions?
summary
Claims that table-heavy ETL is best tested by Cucumber table tests with fast unit checks and SQL formatting via IntelliJ, that pipelines run as Airflow DAGs numbering about 4,000 jobs, and that developers execute PR code on the production cluster writing to ephemeral test tables to validate behavior. Expect 1%–10% sampled or synthetic test data, anomaly models with many false positives, storage cost audits deleting unused tables, and CI/CD on GitHub deploying on green checks.
AI-written summary · about
chapters (12)
- 0:00Testing BigData and ETL
- 3:20What is ETL and pipelines
- 7:58Unit and integration testing
- 12:53Testing DAGs and cycles
- 15:49Test environments for BigData
- 17:35Test pyramid for pipelines
- 19:00Data validation and checks
- 21:46Anomaly detection in pipelines
- 24:17Ownership management for tables
- 26:06Finding bugs via data quality
- 27:10Cost optimization in pipelines
- 28:36Conclusion and key takeaways
related talks
39mWhat Can Go Wrong With Testing an ETL Pipeline?
23mChallenges of database testing
25mTesting Data Pipelines
39mModern Data Pipelines Testing Techniques: A Visual Guide
42mAutomation Testing in a Real Big Data Project
20mBuilding a Data Pipeline with Testing in Mind
45mValidating Big Data Jobs - Stopping Failures before Production
40mPerformance Testing and Big Data