Deciphering the way data is tested
summary
Pipeline validates streaming and batch data end-to-end using Apache Kafka + Spark Streaming for ingestion and real-time checks, stores results in Cassandra, and pushes metrics to Elasticsearch/Logstash/Kibana for dashboards; validations are config-driven (column checks, range and constraint rules, example: product_price > 7000), supports schema/profile discovery, enrichment and monitoring of KPIs like data health, downtime and throughput. Claims 3% of companies meet basic data quality and 80–90% of data is unstructured; demo runs but faces ops friction (Zookeeper startup hiccup). Great Expectations is presented as a lightweight alternative.
AI-written summary · about
chapters (10)
- 0:00Start
- 4:09Data quality dimensions
- 6:25Pre-requisites of data quality testing
- 9:425 steps for better data quality
- 13:21Data pipeline testing
- 17:03Open-source framework
- 33:19Integrating data validation in ML pipelines using tensorflow data validation library
- 34:58Hidden technical debt
- 36:41Challenges and Source of error
- 39:08Monitoring systems
related talks
45mValidating Big Data Jobs - Stopping Failures before Production
40mValidating Big Data Pipelines & ML (w Spark & Beam)
29mData Acquisition: Overcoming Challenges
50mData-Driven Engineering
36mData pipelines engineering made simple with Scala
19mDemystifying Modern Data Pipeline Architecture
20mBuilding a Data Pipeline with Testing in Mind
26mLarge Scale Data Validation (with Spark and Dask)