Building a Data Pipeline with Testing in Mind
summary
Concrete claim: build pipelines that are testable and monitorable using schemas, metadata, and dry runs. Tools used include protobufs for strict schemas, Pachyderm or Airflow/Luigi for orchestration, and Prometheus + Grafana for time‑series metrics and alerts. Example metrics: 95 talks scraped, description length averages 836 characters, gauges for last successful job timestamp and job latency, histograms for items processed vs failed. The system runs end‑to‑end checks and triggers alerts when latency or success rates exceed thresholds.
AI-written summary · about
chapters (13)
- 0:00Start
- 1:25Agenda
- 5:40What Could Go Wrong
- 7:48Core Concerns
- 8:29Interpretability and Observability
- 9:47Observability
- 10:01Monitoring Observability
- 14:40Monitoring Testing Data Pipelines
- 15:38Functional Test
- 15:49Regression Test
- 16:15System Tests
- 16:36Monitoring
- 18:23Reasons Why Data Points Fail
related talks
25mTesting Data Pipelines
19mObservability for Data Pipelines: Monitoring, Alerting & Tracing Lineage
39mModern Data Pipelines Testing Techniques: A Visual Guide
20mShift-left testing : Building reliable Data Pipelines
27mDesigning Your Pipelines for Scalability and Resiliency
37mBuilding a Metrics Optimised Pipeline (Full Automation)
30mHow to Test BigData/ETL Solutions?
45mHow to Build a Performance Testing Pipeline from Scratch