From raw data to predictive model
summary
Wix builds a reusable pipeline that turns raw logs into predictive models: 110 million users, 700 million events/day are aggregated into a single feature store, producing 37,000 → 12,000 → 900 features via filtering, wrapper and embedded selection. Spark handles aggregation and Parquet storage, models use tree methods (XGBoost/gradient boosting) with a train/production split, retrained on a rolling window and served hourly. The claim: one universal dataset plus automated pipeline lets product teams ask new classification questions quickly and get working predictions in production.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 3:31Cross-Industry Process approach
- 5:42Hipster problem example
- 8:55Data collection and prep
- 12:29Data types and features
- 16:44Data agnosticism and universal approach
- 17:14Feature selection methods
- 22:30Modeling and evaluation
- 25:41Deploying to production
- 26:23Apache Spark experience
- 35:12Q&A
related talks
26mFrom Spark to Elasticsearch and Back
01h 18mFrom raw data to interactive data app in an hour
33mProductionizing H2O Models with Apache Spark
41mFrom 5 PB of Daily Logs to Thousands of ML Models #bbuzz
31mLife cycle of data becoming a model
24mJosh Wills, Head of Data Engineering, Slack
27mDeveloping and Deploying Models at Scale — without Spark!
34mPrepare data for machine learning with ease, speed, and accuracy