Declarative Feature Engineering
summary
Adyen builds a Python-based DSL that declaratively defines features and auto-generates Spark and Flink pipelines, metadata, Airflow DAGs and Cassandra/Delta configs; platform handles 40,000 requests per second over 300 terabytes of feature data. It adds a two‑phase partial aggregation optimization to avoid reprocessing and a single backfill job to speed historical fills. Adoption shows three teams using batch features and one testing streaming; the system produces engine-specific SQL and JSON descriptors and is operational with ongoing refinements.
AI-written summary · about
chapters (13)
- 0:00Aden’s ML journey
- 2:15Traditional feature engineering challenges
- 4:22Motivation for DSL
- 6:08Designing the DSL
- 11:45DataFrame API and feature views
- 15:18DSL implementation details
- 18:53Automating feature pipelines
- 20:19Offline vs streaming features
- 21:43Optimizing data processing
- 23:33Speeding up the development cycle for data scientists
- 26:21Adoption and future plans
- 27:03Lessons learned
- 28:32Q&A
related talks
28mData VS Engineers: a long lasting story
18mScaling Feature Engineering with Dagger
01h 28mFeature Engineering Made Simple
20mUsing Flink for Versatile Feature Engineering Pipelines
20mBuilding FeatureFlow, Criteo’s feature data generation platform
33mEvolution and Scaling of Feature Store at Uber
15mFeature Stores: Powering Machine Learning Infrastructure with Apache Flink
23mHow we unified feature engineering across data and backend