Feature Stores: Powering Machine Learning Infrastructure with Apache Flink
summary
Built a production feature store on Apache Flink feeding 10s of millions of users and used for personalization at ~100M MAU, with pipelines on Flink + Pulsar + Cassandra and Pulsar offload to S3. It unifies batch and stream by treating historical events as a stream-s3 ledger so new features can be backfilled instantly, uses Flink temporal tables for point-in-time correct joins, and ships as the StreamSQL feature store that speeds iteration from months to minutes.
AI-written summary · about
chapters (7)
- 0:00Introduction to speaker
- 0:59Machine learning basics
- 2:29Challenges in ML pipelines
- 4:16Key infrastructure problems
- 6:19Using Apache Flink
- 9:24Point-in-time correctness
- 12:59Platform and governance
related talks
31mBuilding an open-source ML feature store with Apache Flink
32mA feature Store for Machine Learning
30mHow Feature Stores Enable Operational ML
08mFeature Store Usage Patterns - From a Single Data Scientist to an Enterprise
20mBuilding FeatureFlow, Criteo’s feature data generation platform
21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & Cassandra
35mPanel 03- How to make the Feature Store become a hidden part of the productionalization of ML models
33mEvolution and Scaling of Feature Store at Uber