What Can Go Wrong With Testing an ETL Pipeline?
summary
Project builds ETL on S3 + Step Functions + EMR Spark jobs with outputs into S3/Redshift/Glue/Tableau; initial test suite has ~5–8 unit tests per metric and end-to-end runs that take about an hour. Team creates a homegrown DSL became spaghetti to generate synthetic examples, discovers maintenance cost outweighs benefits, and rolls back: unit tests use minimal DataFrames, integration tests use mocks to check component flow, and synthetic data is reserved for acceptance tests, making CI faster and cheaper.
AI-written summary · about
chapters (9)
- 0:00ETL pipeline testing intro
- 4:11Data pipeline testing
- 9:49Testing data pipeline areas
- 12:43First testing strategy problems
- 19:22Refining tests with abstractions
- 25:43Synthetic data overuse risks
- 27:27Holistic testing approach
- 31:01Lessons from extreme testing
- 32:47Q&A with Arne
related talks
30mHow to Test BigData/ETL Solutions?
25mTesting Data Pipelines
23mChallenges of database testing
20mShift-left testing : Building reliable Data Pipelines
39mModern Data Pipelines Testing Techniques: A Visual Guide
35mBreaking down large tests with a Saff Squeeze
16mBeyond ETL: Revolutionizing Data Engineering Workflows
20mBuilding a Data Pipeline with Testing in Mind