Streaming your shared ride
summary
Lyft migrates real-time pipelines from Kinesis cron jobs to Kafka + Apache Flink, adds a Python path via Apache Beam so data scientists can reuse model code, and exposes streaming SQL for feature generation; moving pricing from multi-stage Redis crons to a data-driven pipeline cuts orchestration code and instance count and bounds end-to-end latency to ~30s (now dominated by model execution). Operational work includes S3-backed checkpoints, a Kinesis consumer with global watermark synchronization to prevent checkpoint blowup, and a Kubernetes Flink operator for stateful rolling updates.
AI-written summary · about
chapters (27)
- 0:00Intro
- 1:35Data platform users
- 1:55Data Platform architecture
- 2:49PubSub: From Kinesis to Kafka
- 5:09Streaming Compute Stack
- 6:05Flink Abstraction Levels
- 6:56Use Case Categorization
- 9:36Dryft Program Specification
- 9:53Dryft Program Execution
- 10:38Bootstrapping
- 11:42Benefits
- 12:27Dynamic Pricing
- 12:58What is PrimeTime?
- 13:28Legacy architecture: A series of cron jobs
- 14:11Problems
- 14:44Streaming!
- 15:16Multi-Language Support
- 16:10Python Example
- 16:58Python via Beam on Flink
- 18:33Pipeline (conceptual outline)
- 20:10Flink Connectors
- 21:18Challenges
- 24:08Watermark Skew
- 25:16Solution: Source synchronization
- 27:05Contribution to Flink
- 28:38Flink Operator - CRD
- 29:40(Stateful) Upgrade
related talks
28mEnabling Machine Learning with Apache Flink
21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & Cassandra
28mFuture of Data Platforms: making use-case driven...
35mKeynote | Stream Processing at LinkedIn: Then and Now
53mNoLambda: A new architecture combining streaming, ad hoc, machine...
39mStreaming doesn’t have to be hard
25mSharing what we love: The Apache Flink story at Pinterest
44mShaping the Future of Real-time Data Pipeline