Resilient Predictive Data Pipelines
summary
Argues that always-on data pipelines must be resilient, timelier and cheap by design and shows concrete engineering choices: batch on EMR/Spark with hourly runs brought back to ~9 minutes after regressions, transactional fanout using SNS+SQS fanout plus dead‑letter queues to avoid lost messages, autoscaling driven by queue‑depth autoscaling (visible vs invisible count) instead of CPU, workflow authoring and alerts with Apache Airflow, and a cloud schema registry for Avro backed by AWS Lambda to enable elastic stream processing and shard autoscaling.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:33About Me
- 3:21Motivation
- 3:50Data Products
- 6:10Serving + Data Pipelines
- 8:05The Blast Radius Problem
- 12:11Timeliness
- 20:52SOS - Dead Letter Queue
- 22:52SNS + SOS Design Pattern
- 25:02What Does Agari Do?
- 33:22Apache Airflow - Authoring DAGS
- 34:25Apache Airflow - Perf. Insights
- 34:50Apache Airflow - Alerting
- 35:53NRT Architecture
- 39:10Schema Registry
- 40:53What is AWS Lambda?
- 41:54Elastic Stream Processing
- 42:30Open Source Plans
- 42:44SOFTWARE DEVELOPMENT CONFERENCE
related talks
30mDesigning Resilient Data Pipelines
43mData Pipelines à la Mode
45mModern Data Architecture
31mOne Data Pipeline to Rule Them All
45mData pipelines from zero to solid
28mBuilding a Coud Native Data Pipeline for Your Real-Time ML & AI Apps
36mElegant data pipelining with Apache Airflow
24mDistinguishing features of production quality pipelines