Building an open-source ML feature store with Apache Flink
summary
An open source Flink-based framework called Featury implements a protobuf change log written to S3 that is readable both for low-latency online inference and for offline backfills, enabling point-in-time join behavior without a separate online/offline product. It supports counters, bounded lists, maps and quantile estimators, runs the same Scala/Flink code for streaming and batch, processes a year of data for ~1,000 stores in a few hours, and is running in production.
AI-written summary · about
chapters (9)
- 0:00Introduction and speaker background
- 1:08What is Learn to Rank and feature stores
- 7:07Feature logging challenges and feature store solution
- 12:26Why build a custom feature store
- 15:35Apache Flink for feature store
- 17:18Findify’s real-time ML pipeline
- 20:28The do-it-yourself feature store concept
- 23:16Live demo of the feature store
- 27:32Future plans and MetaRank engine
related talks
15mFeature Stores: Powering Machine Learning Infrastructure with Apache Flink
33mFeathr, an Open Source, Battletested, Scalable and Enterprise-Grade Feature Store
30mHow Feature Stores Enable Operational ML
20mUsing Flink for Versatile Feature Engineering Pipelines
35mPanel 03- How to make the Feature Store become a hidden part of the productionalization of ML models
20mBuilding FeatureFlow, Criteo’s feature data generation platform
32mA feature Store for Machine Learning
21mIntroducing FlinkML