Shift-left testing : Building reliable Data Pipelines
summary
Argues that shifting tests left—unit, component, flow and runtime quality checks—catches errors earlier and reduces cost (defects found early are 100x cheaper); cites data downtime losses like $10–14M per org and industry estimates up to $3T. Shows a concrete flight-delay pipeline with unit tests using mock dataframes, component tests writing to local S3 buckets via LocalStack and Spark images, and runtime checks with Great Expectations. Recommends data contracts, logging/monitoring/alerting and anomaly detection to automate recovery and keep pipelines reliable.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:42What is Data Pipeline
- 1:34Data Downtime
- 1:53Does it matter? BAD data is bad for business
- 2:22What can we do?
- 2:36Why Shift Left? Stats from the world of software engineering
- 2:59Before we shift left, lets understand basics Flow of code and data
- 3:54Testing Pyramid
- 6:05Example Tests
- 6:52Unit Test
- 7:34Component Test
- 10:22Local Environment
- 14:42Example Data Quality checks
- 15:39Practical Data test grid
- 16:51Data Contracts
- 18:20Ladder of Reliability
- 18:57Conclusion
related talks
25mTesting Data Pipelines
39mModern Data Pipelines Testing Techniques: A Visual Guide
40mEmbracing Shift-Left Testing
52mShift Left or Squish Right - Testing in a Shifting World
39mWhat Can Go Wrong With Testing an ETL Pipeline?
20mBuilding a Data Pipeline with Testing in Mind
36mShifting Left: How to Build a Culture of Testing
56mTesting at scale