Notebooker: Production and Scheduling for your Jupyter Notebooks
summary
Notebooker converts Jupyter Notebooks into reusable templates that are parametrised reports, executed by Papermill execution and stored in a Mongo-backed archive with permanent permalinks and PDF/HTML rendering. Code-review uses Jupytext/nbdime, scheduling uses APScheduler, and outputs feed Prometheus/Grafana dashboards. Team prototyping takes a few minutes to appear, a few days to prototype and one to two weeks to ship; the app is open-source on GitHub and runs in production at Man Group.
AI-written summary · about
chapters (14)
- 0:00Start
- 1:36Jupyter Notebooks are awful for production!
- 4:24Any alternatives?
- 5:15Notebooker's aims
- 6:07Workflow in 8 steps
- 7:37Reviewing a Notebook - The raw version
- 11:01Introducing Notebooker
- 12:03Parametrisation
- 12:45Publishing / Sharing
- 14:32Scheduling
- 15:08Adding a Scheduler
- 15:59Managing a scheduler
- 19:08Use Case: Data Quality Checking End Result
- 20:03Summary
related talks
39mData and ETL with Notebooks in Papermill
01h 00mFrom notebook to production using
01h 25mTroubleshooting your Data Workflows with Noteable+Dagster-live debugging of failed jobs
33mSignificant Roadblocks to Usefulness for Jupyter Notebooks and a Recipe to Fix them
30mPrototyping Interactive Dashboards with Jupyter Notebooks
27mWriting a Book in Jupyter Notebooks
26mWhat we learned from Papermill to operationalize notebooks
28mUnlock the Full Potential of Jupyter Notebooks