How Lyft built a streaming data platform with Flink on Kubernetes
summary
Lyft moves Flink from EC2 to Kubernetes and ships an open-source Flink Kubernetes operator that encodes deployment logic: it creates a fresh cluster, takes savepoints, cancels the old job, submits the new job and rolls back on failure. The platform runs 80+ jobs by 50+ engineers processing tens to hundreds of millions events/sec, supports Beam/Python and Dryft SQL, cuts cluster provisioning to new cluster per deploy <2 minutes, and is now the primary deployment model.
AI-written summary · about
chapters (7)
- 0:00Introduction and overview
- 3:29Why Flink matters at Lyft
- 8:10Legacy Flink deployment challenges
- 16:37What is Kubernetes
- 19:57Why build a Flink operator
- 25:53Operator lifecycle management
- 31:12Challenges and next steps
related talks
19mLyft and the California Consumer Privacy Act
40mFlink in Zalando's World of Microservices
38mTaking away customer friction through streaming analytics
42mRunning a Massively Parallel Self-Serve Distributed Data System at Scale
45mDeep dive into Unbounded Data Processing Systems
20mBatching vs. Streaming - Scale & Process Millions of Measurements a Second
51mStreaming Platforms in Event-Driven Systems: Patterns, Pitfalls, and Performance
25mKeynote: Cloud Native Updates