Stream Processing at Lyft with Flink and Beam

on YouTube ↗32m

" summary

Lyft runs real-time pipelines using Apache Flink as the runner and is adopting Apache Beam for Python portability, operating 50+ Flink jobs processing tens of millions of records per second on EC2 (Kubernetes work in progress). Bootstrapping reads historic S3 data plus live Kinesis, creates a checkpoint snapshot, then restarts at lower parallelism; a large fraud job (60 days of events) bootstraps in about four hours. A custom bounded priority queue plus a global watermark (ZooKeeper) prevents partition skew and reduces checkpoint/backpressure failures.

AI-written summary · about

" chapters (8)

1. 0:00Introduction to Lyft streaming
2. 3:27Batch vs stream processing
3. 5:14Stream processing concepts
4. 10:30Apache Flink at Lyft
5. 13:21Apache Beam for Python support
6. 14:06Bootstrapping stream jobs
7. 19:51Source queue challenges
8. 25:54Q&A session

" related talks

• 40mLarge-scale near-real-time (NRT) data analytics platform empowered by Apache FlinkYing Xu, Kailash HD · Flink Forward Virtual 2020
• 19mBuilding Stateful Streaming PipelinesAnkit Jhalaria · Flink Forward Virtual 2020
• 34mBeaming Flink to the Cloud @ NetflixMonal Daxini · Flink Forward Berlin 2016
• 33mFlink and Beam: Current State & RoadmapMaximilian Michels · Flink Forward Berlin 2016
• 36mData processing at Spotify - Why portability mattersJulien Tournay, Filipe Regadas · Flink Forward Global Virtual 2020
• 30mBuild and run streaming applications with Apache Flink and Amazon Kinesis Data AnalyticsHausmann · Flink Forward Europe 2019
• 20mScale By The Bay 2019: Sharanya Santhanam & Shiv ToolsidassSharanya Santhanam, Shiv Toolsidass · Scale By The Bay 2019
• 40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobsSteffen Hausmann, Danny Cranmer · Flink Forward Global 2021