Data pipelines from zero to solid
summary
Recommendations and concrete practices for building reliable data pipelines: use Kafka unified log for ingestion, mirror to HDFS as an immutable data lake and seal hourly datasets; store schema with Avro/Parquet; prefer batch over streams unless latency demands real time; process with Spark + Scala for scale and fast local iteration; orchestrate and backfill with Luigi workflow manager; serve denormalized results in Cassandra; test full jobs locally and log code commit hashes; note a real outage: a Cassandra export once grew to 25 hours and broke logins, so keep ingestion simple and immutable.
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:59Presentation goals
- 2:47Data product anatomy
- 3:41Computer program anatomy
- 6:35Event registration
- 7:16Event transportation
- 10:11Anti-pattern: Send the oliphants!
- 11:55Using snapshots
- 14:47The data lake
- 16:36Representation - data lake & pipes
- 23:27Batch processing
- 24:18Batch job code
- 25:12Language choice
- 29:56Workflow manager
- 32:33Egress datasets
- 34:17Test strategy considerations
- 35:42Testing single job
- 36:35Testing pipelines - two options
- 39:08Continuous deployment
- 40:40Protecting privacy in practice
- 43:06Simple Pll audit
- 44:17Parting words + sales plug
related talks
31mOne Data Pipeline to Rule Them All
42mBuilding Robust Streaming Data Pipelines with Apache Spark
24mDistinguishing features of production quality pipelines
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka
40mTop 10 data engineering mistakes #bbuzz
45mResilient Predictive Data Pipelines
39mUsing Streaming Data Pipelines in the Cloud to Achieve Success
43mData Pipelines à la Mode