Data Pipeline Evolution
summary
Describes evolving a pipeline at FanDuel from slow daily full-table ETL to mixed incremental batches and near-real-time events, implemented in Python using Amazon EMR, Redshift, Hadoop, Luigi, Spark and Kinesis. Shows concrete changes: full daily imports taking hours, then incremental ID/timestamp-based loads, Spark for heavier transforms, Kinesis for event streams, and experiments with Glue/Athena to cut costs. The hybrid system reduces latency and scales better but introduces latency mismatches, duplication risks and ongoing cost/operational trade-offs.
AI-written summary · about
chapters (17)
- 0:00Start
- 0:28Data Pipeline Evolution
- 1:01Who Am I
- 4:46What a Big Data Engineer Does
- 8:32The Positives of Using a Software Product for Data Warehousing
- 9:06Pricing
- 10:54Technologies
- 11:33Redshift
- 12:44Advantages of the Full Import Approach
- 13:54Games and Entries
- 15:26Apache Spark
- 15:56Advantages of Bringing In Data by the Batch
- 17:58The Incremental Import Model
- 19:59Lambda Architecture
- 20:58Challenge of Building and Maintaining a Distributed Data Pipeline
- 29:59Questions
- 31:00How Do I Manage the Schema
related talks
31mOne Data Pipeline to Rule Them All
30mData Engineering building your BI infrastructure from scratch
29mEvent streaming with modern data pipelines in a SaaS architecture
45mModern Data Architecture
29mSimplifying data flows
25mA simple serverless data pipeline
43mData warehousing reinvented for today's needs
34mTales of Data Architecture Evolution