Using Flink for Versatile Feature Engineering Pipelines
summary
Deep.BI builds a Flink-based feature-engineering framework that turns raw event streams into reusable features and live predictions with under one second latency, processing ~40 billion rows per month and peaks near 100k events/sec. It uses the low-level DataStream API, control streams and Broadcasted State to push Python/JavaScript filters and serve TensorFlow/PFA/CatBoost models, maintains short- and long‑TTL state, provides unit test harnesses, and updates feature logic live with near‑zero downtime.
AI-written summary · about
chapters (8)
- 0:00Introduction and speaker intro
- 1:56Why feature engineering matters
- 3:43Platform architecture overview
- 5:13Design choices for Flink pipelines
- 8:49Testing feature engineering code
- 10:14Updating feature state at scale
- 12:46Live demo of feature updates
- 19:47Summary and Q&A
related talks
21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & Cassandra
36mDeclarative Feature Engineering
31mBuilding an open-source ML feature store with Apache Flink
20mBuilding FeatureFlow, Criteo’s feature data generation platform
37mBuild a Flink AI Ecosystem
18mScaling Feature Engineering with Dagger
25mData: Moving with the flow
18mMachine learning with Flink in Weibo