Going From Notebooks to Scalable Systems
summary
Shows exactly what gets changed: convert a penguin-classification notebook using scikit-learn into modular Python modules, extract steps (download, clean, preprocess, train, predict), and wrap them in a single pipeline that saves artifacts and can retrain automatically. Uses nbconvert and jupytext to export code, trains a logistic regression on the Palmer Penguins dataset with an 80/20 split after scaling and label encoding, and adds unit tests and CI/CD to catch copy‑paste/state bugs and make the system reproducible.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 2:29Why move to scalable systems
- 7:04Notebook to scalable code
- 12:49Refactoring notebook code
- 18:55Testing modular code
- 21:21Best practices for scalable code
- 23:33Q&A
related talks
01h 28mMaintainable code in data science
27mNotebook To Production
36mThe Lifecycle of a Jupyter Environment
33mUsing Scikit-Learn's interface for turning Spaghetti Data Science into Maintainable Software
01h 33mFrom Raw Recruit Scripts to Perfect Python
28mUnlock the Full Potential of Jupyter Notebooks
01h 09mNotebook to Pipeline: Hands-On Data Engineering w Python
27mThe necessity of pipelines in Natural Language Processing