Reproducible and shareable notebooks across a data science team
summary
CybelAngel processes about a billion candidate leaks and is scaling a data team that plans to double; notebooks drive exploration but were hard to share, version and reproduce. The solution runs managed VMs on GCP with JupyterHub for per-user sessions authenticated via GitHub, pairs notebooks with text files using Jupytext so edits serialize to a bi-directional py/ipynb sync, stores code in Git and packages with Poetry, which removes duplicated copy‑paste, improves traceability and speeds handoff to scripted production.
AI-written summary · about
chapters (9)
- 0:00Start
- 3:37Example of Sensitive Data
- 5:15Traceability and Reproducibility
- 5:58Jupyter Notebook
- 7:22Machine Learning Workflow
- 10:00Drawbacks
- 15:38Limitation Is Traceability and Reproducibility
- 21:27Clone a New Project
- 27:49Key Takeaways
related talks
25mOvercoming 5 Hurdles to Using Jupyter Notebooks for Data Science, by the JetBrai...
28mCollaborate with your team using data science notebooks
34mMoving notebooks into the cloud: challenges and lessons learned
33mSignificant Roadblocks to Usefulness for Jupyter Notebooks and a Recipe to Fix them
28mHosting Notebooks for 100,000 Users
25mTo Notebook or Not to Notebook: Multilingual Workflows for Data Analysis
39mEnabling Data centric Teams
49mExperimenting with Large Data Volumes Using Notebooks