Creating custom engines for flexible notebook execution
summary
Shows how to build and register papermill engines as Python entry points, with an execute method that replaces the default subprocess model. Demonstrates a profiling engine using mprof to reveal memory bumps when allocating large numpy arrays, a debuglater engine that serializes interpreter state to a jupyter.dump for remote postmortem debugging, and a plumber experiment tracker that logs ~45 runs into SQLite (best F1 ≈ 0.8) enabling SQL queries and plots.
AI-written summary · about
chapters (10)
- 0:00Custom Paper Mill engines
- 2:14Extending Paper Mill
- 5:06Notebook profiling
- 7:40Remote debugging
- 9:46Experiment tracking
- 11:13Notebook profiling demo
- 14:02Debugging demo
- 17:59Tracking demo
- 22:44Beyond Paper Mill
- 23:48Q&A
related talks
32mCombining IPython with Open Source Papermill to enhance your Jupyter Notebook
26mWhat we learned from Papermill to operationalize notebooks
01h 23mImproving Your Data Modeling Work Through Open-Source Software
34mNotebooker: Production and Scheduling for your Jupyter Notebooks
27mAdding interactivity by extending Jupyter Notebook
33mSignificant Roadblocks to Usefulness for Jupyter Notebooks and a Recipe to Fix them
01h 44mLeveling up your Jupyter notebook skills
36mThe Lifecycle of a Jupyter Environment