Moving notebooks into the cloud: challenges and lessons learned
summary
Civis Analytics claims a working cloud Jupyter integration that runs notebooks in Docker on Kubernetes, fronts them with an auth proxy issuing 10‑second rotating URLs, and routes with the open‑source tool traffic. It ties notebooks to Redshift and an always‑on query store to support both reproducible and live reports, uses per‑client isolation to limit blast radius, and ships an MVP with 32GB nodes where notebooks default to 2GB but can scale up; the system is built, demoed, and collecting user feedback.
AI-written summary · about
chapters (9)
- 0:00Cloud notebook integration
- 2:29Collaboration challenges
- 4:57Notebook integration problem
- 6:23Design sprint to prototype
- 8:08User research and requirements
- 12:47Engineering process and MVP
- 17:02Architecture and security
- 27:52User feedback and iteration
- 33:59Q&A
related talks
32mSpin up Jupyter notebooks at scale and enhance your productivity
37mReproducible and shareable notebooks across a data science team
49mExperimenting with Large Data Volumes Using Notebooks
39mEnabling Data centric Teams
28mHosting Notebooks for 100,000 Users
25mOvercoming 5 Hurdles to Using Jupyter Notebooks for Data Science, by the JetBrai...
12m⚡ Lightning Talk: Data Science Environments in Seconds: Scaling Jupyter Notebooks in Kubernetes
30mOnyxia: A User-Centric Interface for Data Scientists in the Cloud Age