Data VS Engineers: a long lasting story
summary
A homegrown Data Feature Store called Efesto Data Feature Store standardizes feature engineering with Python and PySpark libraries (fl_clon, fto_core, F features), uses Airflow + Dataproc serverless + BigQuery for orchestration and storage, and enforces tests via DBT data quality gates. It implements hash-based historicization to keep latest, snapshot and full history tables, exposes dependencies so Airflow orders jobs, and runs in production with ~70 developers, 400 features across 18 DAGs and ~100k CPU hours per month.
AI-written summary · about
chapters (12)
- 0:00Introduction to data challenge
- 1:08Theoretical framework overview
- 3:38Festo feature store architecture
- 5:05Feature group and feature classes
- 6:52Coding example setup
- 10:52Business logic implementation
- 12:43Historical data management
- 15:12Data quality with DBT
- 16:42Airflow integration details
- 20:18DAG deployment process
- 22:34Usage statistics and impact
- 23:22Q&A session
related talks
36mDeclarative Feature Engineering
30mDataEngBytes 2023 - MEL-T2-01 - Mikiko Bazeley
38mScaling Data Science
08mFeature Store Usage Patterns - From a Single Data Scientist to an Enterprise
10mEmpowering your feature stores with AI feature discovery
33mFeathr, an Open Source, Battletested, Scalable and Enterprise-Grade Feature Store
30mHow Feature Stores Enable Operational ML
32mA feature Store for Machine Learning