Eventually, time will kill your data processing
summary
Lists concrete failure modes — daylight savings, leap seconds, client clocks, late mobile events, and duplicated hours — and shows where they break data collection, batching, windowed joins and sessionization. Recommends practical fixes: use Airflow or Luigi for explicit orchestration, mark completed ingest buckets with _SUCCESS at ingest time, choose wait thresholds (Spotify learned to reprocess on 1–2% late arrivals is an anti‑pattern), and prefer careful windowing or bounded caps rather than blind streaming.
AI-written summary · about
chapters (29)
- 0:00Introduction
- 5:12Data Processing
- 7:46Time
- 11:33Technical systems
- 12:58Snapshots
- 14:30Database Dump
- 14:58Event Sourcing
- 17:12Windows over something
- 18:15Window of time
- 19:25Batch processing
- 20:00Collecting events
- 20:47Late events
- 21:48Tools for reprocessing
- 22:43Use one bucket
- 23:12Data sets
- 23:45Downstream decision
- 24:19Workflow orchestration
- 25:38downstream consumers
- 26:19additional information
- 26:52late data
- 27:23time scopes
- 28:17time dependencies
- 29:11quick strides
- 29:46backfill
- 31:10artificial cap
- 32:20general advice
- 33:51team parallelization
- 34:16data engineering
- 34:46technology and society
related talks
31mWhen the data levee breaks: Managing cognitive load in data-intensive projects
37mAuditing your data and answering the question, is it the end of the day yet?
25mData Demons: Unleashing Chaos or Driving Performance?
31mZero to data infrastructure
17mOptimizing Data-Driven Decisions
41mFast Data - an Overview of the Concepts, Architecture and Technology of Streaming Analytics
42mHow I learned to time travel, or, data pipelining and scheduling with Airflow
41mTreading Water in a Stream of Data