Productionizing Jupyter Notebooks
summary
Demos a workflow that turns exploratory Jupyter notebooks into runnable data jobs using Versatile Data Kit (VDK): notebooks are tagged as VDK sales to enforce a strict execution order, code runs locally with ingestion into SQLite, and tests run end-to-end with a pdk run command before deployment. The notebook-to-pipeline flow preserves reproducibility, strips transient outputs on deploy, supports plugins and one‑click deployment, and surfaces issues like a counted 365 test users that break loose cell execution.
AI-written summary · about
chapters (13)
- 0:00Productionizing Jupyter Notebooks
- 1:09Challenges in productionizing
- 2:38Verstyle Data Kit overview
- 11:32Workshop environment setup
- 19:08Reproducibility issues in notebooks
- 24:18Reproducibility with VDK cells
- 33:53Marking cells for production
- 43:41Debugging and visualization cells
- 53:38Data ingestion with VDK jobs
- 58:22Testing notebook logic
- 1:01:13Cleaning notebooks for deployment
- 1:04:43VDK common questions
- 1:13:51Closing and survey
related talks
01h 25mTroubleshooting your Data Workflows with Noteable+Dagster-live debugging of failed jobs
26mTransforming a Jupyter Notebook into a reproducible pipeline for ML experiments
49mTutorial: From Notebook to Kubeflow Pipelines to KFServing: the Data Science Odyssey
28mDevOps for Data Science With Kubernetes
39mNotebook To Production
01h 27mTutorial: From Notebook to Kubeflow Pipel...
53mTransforming your Jupyter Notebook to a Reproducible Machine Learning Pipeline using Kale
22mDock Your Jupyter Notebook