Maintainable code in data science
summary
Notebooks refactor into a testable Python package that builds a full pipeline: custom preprocessing classes that memorize state, a ColumnTransformer + Pipeline combining PCA/one‑hot and a classifier, GridSearchCV that tunes both preprocessing and model, and joblib persistence to save a single serialised pipeline. Exercises on GitHub provide the code and notebooks, the pipeline trains, cross‑validates and loads correctly, and examples show how many parameters to tune and how to avoid leakage.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:14Defining maintainable code
- 2:57Data science project stages
- 5:49Maintainable feature engineering
- 10:04Scikit-learn transformers
- 15:24Exercise environment
- 23:12Kickstarter dataset
- 27:54Custom transformers
- 33:13Category extractor transformer
- 43:26ColumnTransformer advanced
- 48:10Modular preprocessing pipelines
- 51:42Exercise: preprocessing pipeline
- 58:46Building the preprocessing pipeline
- 1:04:57Training and tuning model
- 1:09:12Saving and loading models
- 1:17:19Testing new data
- 1:20:47Model management CLI
- 1:23:01Maintainable code best practices
- 1:25:31Q&A
related talks
28mGoing From Notebooks to Scalable Systems
33mUsing Scikit-Learn's interface for turning Spaghetti Data Science into Maintainable Software
01h 09mNotebook to Pipeline: Hands-On Data Engineering w Python
41mBetter Code for Data Science
36mThe Lifecycle of a Jupyter Environment
27mThe necessity of pipelines in Natural Language Processing
26mReproducible and maintainable data science code with Kedro
01h 14mUp your Bus Number - A Reproducible Data Science Workflow