Enabling Machine Learning with Apache Flink

on YouTube ↗28m

" summary

Covers how Lyft built a real-time feature-generation ML platform using Apache Flink Streaming for ride-sharing decisions (e.g., matching drivers to passengers). Describes concrete engineering challenges and solutions: handling skew when reading from multiple sources, throttling throughput when calling microservices, bootstrapping pipelines, and operational practices for production-quality ML in Flink.

AI-written summary · about

" chapters (16)

1. 0:00Start
2. 4:02User action sequence
3. 6:21What Data Scientists care about
4. 6:56ML Workflow
5. 7:51Needs
6. 8:31Dryft! - Self Service Streaming Framework
7. 9:42Job Definition
8. 11:13Feature Fanout App - also uses Dryft
9. 13:09One Job Manager to Rule Them All
10. 13:33Future is ... Flink on Kubernetes
11. 15:14Custom Resource Descriptor
12. 16:16What is bootstrapping?
13. 17:10Bootstrap with historic data
14. 17:58Solution - Consume from two sources
15. 20:29Output volume spike during bootstrapping
16. 22:33Solution: Source synchronization

" related talks

• 37mStreaming your shared rideThomas Weise · Berlin Buzzwords 2019
• 15mData science at scale using Apache Flink: Dynamic model serving and real-time feature generationShashank Agarwal · Flink Forward Global Virtual 2020
• 28mFuture of Data Platforms: making use-case driven...Ketan Umare · Global Artificial Intelligence Conference 2020
• 16mAnalysing Real time Activity Streams for SecurityRashmi Singh · Berlin Buzzwords 2019
• 40mStreaming ML with FlinkMarton Balassi · Flink Forward Berlin 2016
• 15mFeature Stores: Powering Machine Learning Infrastructure with Apache FlinkSimba Khadder · Flink Forward Global Virtual 2020
• 31mLeveraging GraphQL for Continual Learning in Real-Time ML SystemsAryan Magoon · GraphQL Summit 2023
• 37mMoving People and Products with Machine Learning on KubeflowWillem Pienaar, Jeremy Lewi · KubeCon + CloudNativeCon Europe 2019