The Lego Model for Machine Learning
summary
A machine learning platform called Optimus Prime built on Spark + Scala generates code to automate a ten‑step feature engineering pipeline: feature typing, regex extraction, missing‑value imputation, normalization and bucketization, interaction and augmented features, and sanity checks for label leakage and multicollinearity. It tries many model families (logistic, random forest, hyperparameter grids), turns training sets with thousands of candidate features into models with dozens of predictors, and passes benchmarks on a couple dozen datasets, producing deployable per‑customer models.
AI-written summary · about
chapters (12)
- 0:00Introduction and context
- 4:21Why one-size-fits-all ML fails
- 7:36Introducing Optimus Prime
- 10:43Feature classification and typing
- 14:56Handling missing values
- 17:48Transformations and normalization
- 23:13Augmented features and data enrichment
- 26:03Interaction features
- 28:32Label leakage and pitfalls
- 31:20Resampling and model selection
- 33:43Summary and key takeaways
- 35:30Q&A session
related talks
34mComplex Machine Learning Pipelines Made Easy
38mFantastic ML apps and how to build them
37mMachine Learning, The Right Way
49mImpact on business and society
39mMachine Learning with Scala on Spark
37mAutomated Machine Learning: Building a Pipeline
30mOperationalizing Machine Learning
23mRethinking scalable machine learning with Spark ecosystem