Data Lake as Source to Apache Flink to Better Support Machine Learning Use Cases
summary
A custom Flink source reads Parquet files in S3 using the dlsync metadata stream as an index and parquet file helpers to track per-file progress. A Dual Source Consumer wraps Kafka and S3 so jobs can backfill terabytes and enable a two‑year backfill while reusing the same Flink logic. The implementation uses Flink’s global aggregate manager for cross‑subtask coordination and a global watermark to prevent skew; a skewed benchmark backfill falls from ~2.5 hours to 37 minutes.
AI-written summary · about
chapters (18)
- 0:00Intro
- 1:40What is Stream Processing?
- 2:53An ML Use Case: Bot Detection in Real Time
- 3:35Yelp's Data Pipeline Ecosystem
- 4:29Kafka: Limited Retention
- 6:12High Level Architectural Options
- 7:37Yelp's Data Lake
- 9:17Data Lake State Topic: The Data Part
- 10:56The Dual Source Consumer
- 11:26Data Lake Source Extends
- 12:40Dual Source Extends
- 13:39Data Lake Source Initialization
- 14:16Data Lake Source Run Method
- 15:56Consumer State
- 16:38Dual Source Summary
- 23:09Watermark Skew Grows
- 28:15Challenge #4
- 31:40Acknowledgements
related talks
15mFeature Stores: Powering Machine Learning Infrastructure with Apache Flink
27mData enrichment patterns with Apache Flink
21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & Cassandra
39mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
33mBridging Big Data and Machine Learning Ecosystems: A Cloud Native Approac...
31mBuilding an open-source ML feature store with Apache Flink
25mSharing what we love: The Apache Flink story at Pinterest
09mFrom Factory Floor to Cloud — Scaling Manufacturing Data on