Large-scale near-real-time (NRT) data analytics platform empowered by Apache Flink
summary
Lyft builds a Flink-powered ingest pipeline that pulls from Kinesis, aggregates into micro-batches, and writes columnar files to AWS S3 with multipart commits tied to Flink checkpoints, claiming p99 freshness under 5 minutes while handling hundreds of billions of events per day. The stack uses Apache Flink for streaming persistence, Apache Airflow for compaction/dedup and Hive metastore atomic partition swaps, and serves interactive queries via Presto. Small-file explosion is cut by ~40x with event-aware partitioning; recovery can spike to ~8–9 minutes and required isolated jobs and schema harmonization.
AI-written summary · about
chapters (21)
- 0:00Intro
- 1:56Streaming data scenarios at Lyft
- 3:07Lyft's data analytics platform architecture
- 5:29Issues of the legacy platform
- 8:12Near real-time data analytics platform architecture
- 9:49Platform Design
- 12:11Platform Characteristics and use cases
- 13:48Flink empowered near real time data ingestion
- 18:30ETL Multi-tier compaction and deduplication
- 19:46Event-time driven partition sensing - Flink
- 24:32Event-time driven partition sensing: ETL
- 25:50Deduplicate Records
- 27:0553 Layout and Partition Management
- 28:2553 File Sensor - Downstream ETL
- 30:05Data backfilling: coping with failures and
- 32:44Challenges for Migration
- 34:16Design of Migrating Schemas
- 35:17Migrating Existing workload (Run every day)
- 36:15Experience and Lessons Learned
- 38:00Future Directions
- 39:15Teamwork and Acknowledgements
related talks
32mStream Processing at Lyft with Flink and Beam
20mScale By The Bay 2019: Sharanya Santhanam & Shiv Toolsidass
30mBuild and run streaming applications with Apache Flink and Amazon Kinesis Data Analytics
04mInterview
40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobs
45mApache Flink — Real-Time Data Processing
34mBeaming Flink to the Cloud @ Netflix
46mSmart Transit: Real-Time Transit Information with FLaNK