Improving Your Data Modeling Work Through Open-Source Software
summary
Shows concrete tools and outcomes: parameterized notebooks run as a 45‑experiment grid and produce per‑run pickles and reports; a CLI called debuglater attaches to failed notebook cells so production failures are replayable and fixable; memory profiling uses a custom executor (contrasted with Paper Mill which hides per‑cell memory) and surfaces where RAM spikes; experiment outputs are indexed into a notebook database (SQLite) queriable by SQL; cloud execution scales the same workflow to many machines.
AI-written summary · about
chapters (10)
- 0:00Open-source data modeling tools
- 5:12Workshop environment setup
- 6:06Parallelizing model training
- 23:26Debugging notebooks
- 29:54Profiling notebooks
- 34:50Monitoring environments Q&A
- 51:23Analyzing experiment results
- 1:00:17SQL-based notebook analysis
- 1:13:25Cloud experiments with Bloomberg Cloud
- 1:20:11Future features & Q&A
related talks
36mThe Lifecycle of a Jupyter Environment
28mUnlock the Full Potential of Jupyter Notebooks
39mEnabling Data centric Teams
33mSignificant Roadblocks to Usefulness for Jupyter Notebooks and a Recipe to Fix them
17mProductionizing Data Science Workloads
27mNotebook To Production
34mData Products, From 0 to data science pipeline
49mExperimenting with Large Data Volumes Using Notebooks