Building FeatureFlow, Criteo’s feature data generation platform
summary
FeatureFlow is a hosted Flink SQL platform that generates short‑term features for Criteo’s scale (~300 billion bids / ~4 billion displays per day) by running jobs on Mesos, reading Kafka, and writing to HDFS, Couchbase or Kafka. Users submit SQL plus UDFs, iterate with a CLI and validate with JSON end‑to‑end tests. Contributions include Azure blob checkpoints and Mesos scheduler knobs that cut scheduling from hours to minutes, Row Serializer and projection pushdown giving 5–8x and 2.5x speedups, and a lifecycle fix that restores checkpoints from ~30s to ~2–3s. FeatureFlow
AI-written summary · about
chapters (9)
- 0:00Introduction and motivation
- 3:28Why build FeatureFlow
- 7:41Architecture overview
- 10:28User interface and DSL
- 12:54Testing and validation
- 14:21Scheduler and deployment
- 15:01Mesos runtime challenges
- 18:49Sources and sinks performance
- 19:31Future work and Q&A
related talks
15mFeature Stores: Powering Machine Learning Infrastructure with Apache Flink
36mDeclarative Feature Engineering
20mUsing Flink for Versatile Feature Engineering Pipelines
15mData science at scale using Apache Flink: Dynamic model serving and real-time feature generation
31mBuilding an open-source ML feature store with Apache Flink
15mOn-demand low-latency feature generation at Hyperconnect
33mFeathr, an Open Source, Battletested, Scalable and Enterprise-Grade Feature Store
18mScaling Feature Engineering with Dagger