Horizontal Data Freshness Monitoring in Complex Pipelines
summary
Google builds a curated data dependency graph from harvested access logs, metadata (mod time, writer, TTL, logical size) and distributed traces, then runs horizontal freshness monitoring that auto-infers age thresholds and flags slow bleeds without direct data reads. The prototype uses centralized log harvesting plus open-source collectors, scales with minimal config, and catches real slow bleeds and surprising unused “absolute” copies of data; tuning (scoping, duplicate exclusion, auto-tuning) improves signal-to-noise.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:58Common challenges many organisations face
- 2:19Implications on data integrity
- 2:57Risk multiplies with each hop downstream
- 4:07Mapping data dependencies automatically
- 4:57Recording data access events Levels
- 5:50Mapping a data dependency graph
- 7:12Instrumenting centralized data access recording
- 8:23Continuous data annotation
- 9:42Plethora of Di solutions to build on top
- 10:40Slow bleed problem
- 12:15Freshness monitoring at scale
- 13:39Types of detectable data anomalies
- 15:47Key features
- 17:04One of Google's solutions: breakdown
- 18:49Lessons learned
related talks
30mDesigning Resilient Data Pipelines
38mEventually, time will kill your data pipeline
20mTime Travel and Provenance for Machine Learning Pipelines
31mWhen the data levee breaks: Managing cognitive load in data-intensive projects
42mIngestion at Scale: Open Source Pipelines for Getting Data Into Open...
41mMonitoring Big Data Systems Done "The Simple Way"
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
19mBuilding Stateful Streaming Pipelines