Using Scikit-Learn's interface for turning Spaghetti Data Science into Maintainable Software
summary
Shows how Scikit-Learn’s estimator/meta-estimator API turns preprocessing into small composable units: implement a Transformer, discover a bug when NaNs break stddev, fix it by splitting into two steps and compose with make_pipeline using Imputer + StandardScaler, then attach the estimator so the whole pipeline is serializable with pickle. Emphasizes testing each Transformer with unit tests (pytest) and using the fit/transform separation so predict-time data gets identical preprocessing; mentions grid search and cross‑validation for model validation.
AI-written summary · about
chapters (8)
- 0:00Introduction to scikit-learn
- 1:02Supervised learning workflow
- 4:28Data preprocessing techniques
- 10:20Transformers and consistency
- 13:40Pipeline and modular design
- 17:28Composing modular pipelines
- 20:51Software engineering for ML
- 24:36Q&A and closing remarks
related talks
27mSoftware Library APIs: Lessons Learned from scikit-learn
01h 28mMaintainable code in data science
43mDeploying Machine Learning using sklearn pipelines
42mWriting a scikit-learn compatible estimator in the modern age
35mPandas, Pipelines, and Custom Transformers
01h 24mInpsect and try to interpret your scikit-learn machine-learning models
27mThe necessity of pipelines in Natural Language Processing
32mExtending Spark Machine Learning Beyond Linear Regression