Eventually, time will kill your data pipeline
summary
Catalogues concrete failure modes (daylight saving causing silent hourly data loss, leap second spikes, time-zone offsets) and shows reproducible fixes: using ingestion time buckets to close datasets, the loading dock pattern that caused missed hourly files, and when to prefer event sourcing versus joined snapshots. Explains windowing trade-offs (processing vs event time), a pragmatic delay-and-count strategy for late arrivals, multi-pass (lambda-style) pipelines for accuracy, and operational mitigations like recursive jump checkpoints (think I-frame/P-frame style backfills).
AI-written summary · about
chapters (33)
- 0:00Intro
- 0:40Out of joint
- 2:13Data categories, time angle
- 3:56Clocks
- 4:52Calendars, naive
- 6:24Calendar reality
- 8:03Small problems in practice
- 8:18Typical loading dock ingress
- 10:16Batch job - functional principles
- 11:22Database dumping
- 12:01Pattern: Offline replica
- 12:24Using snapshots
- 13:51Window misalign
- 14:29Event sourcing
- 16:21Stream window operations
- 16:51Window over what?
- 20:27Anti-pattern: Reprocess
- 20:45Supporting reprocessing
- 21:58Pattern: Ingest time bucketing
- 22:21When data is late
- 22:57Incompleteness recovery
- 24:00Fast data, complete data
- 24:27Lambda in batch
- 26:02Dual time scopes
- 26:50Recursive dependencies
- 27:49Recursive dependency strides
- 28:27Business logic with history
- 28:56What is a session?
- 29:52Occurrences with unbounded time spans
- 30:50Secrets of valuable data engineering
- 31:42Team concurrency
- 33:18The real value of big data
- 33:47Resources, credits
related talks
30mDesigning Resilient Data Pipelines
45mResilient Predictive Data Pipelines
31mWhen the data levee breaks: Managing cognitive load in data-intensive projects
45mData pipelines from zero to solid
44mPreventing, Diagnosing & Curing Bad Data
43mData Pipelines à la Mode
21mSoftware Best Practices for Data Pipelines: Form Energy's Multi-Day Storage R&D Pipeline 🌳
38mTime-Based Aggregates: The Good, the Bad and the Ugly