Up your Bus Number - A Reproducible Data Science Workflow
summary
Claim: a reproducible end-to-end data science workflow that makes the messy 80% of data munging explicit and recoverable. Tools actually used include cookiecutter template, git + github/gitlab, conda or pipenv, Jupyter notebooks, scikit‑learn pipelines, joblib pickles, makefiles for targets like make raw / make data, PathLib, and optional S3 for raw archives. The repo builds dataset objects (raw → interim → processed), trains and serializes models, records hashes and licenses, and the demo runs successfully for most attendees.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 2:16What is a bus number
- 7:41Why reproducible matters
- 15:55Core tools for workflows
- 24:48Interactive tutorial
- 34:19Break
- 36:01Make files in workflows
- 40:09Raw data handling
- 53:56Processing raw data
- 1:06:23Automating data munging
- 1:13:26Training models
- 1:14:09Q&A
related talks
01h 28mHacking the Data Science Challenge
01h 28mMaintainable code in data science
38mBuilding Data Pipelines in Python
20mSkrub DataOps, Probabl
26mReproducible and maintainable data science code with Kedro
35mYou got your engineering in my Data Science: Addressing the reproducibility crisis
01h 19mDatascience on the web
37mPatterns for Collaboration Between Data Scientists and Software Engineers