On-demand low-latency feature generation at Hyperconnect
summary
Hyperconnect builds a Flink-based real-time feature store (three layers: User Event Store, Feature Generator, Feature Consumer) that serves on-demand features to online models and an event generator; it ingests Kafka events for a product with ~180M discoveries/day, stores timelines, supports add/update/delete feature definitions, and emits metrics to Elasticsearch. The pipeline uses timers for delayed triggers (example: 5-minute delayed events), applies beta-distribution smoothing for early estimates, monitors latency and anomalies, and supplies feature vectors and thresholded events to downstream systems.
AI-written summary · about
chapters (9)
- 0:00Introduction to team and talk
- 0:30Motivation for real-time features
- 3:05System architecture overview
- 4:07Event store design and components
- 8:39Aggregating values and windowing
- 11:12Feature generator components
- 12:39Feature consumer use cases
- 13:00Operational lessons learned
- 14:08Summary and conclusion
related talks
23mData driven matchmaking streaming at HyperConnect
20mBuilding FeatureFlow, Criteo’s feature data generation platform
40mCreating and operating ML models from event based data using feature stores
31mBuilding an open-source ML feature store with Apache Flink
32mA feature Store for Machine Learning
36mDeclarative Feature Engineering
25mBuilding A Feature Store For Hyper Growth
31mLeveraging GraphQL for Continual Learning in Real-Time ML Systems