Extending Jupyter with Google Cloud Storage file system backend
summary
Concrete implementation of a Jupyter filesystem backend for Google Cloud Storage that replaces the default disk-based ContentManager, exposes buckets and prefixes as notebooks and folders, and implements checkpoints and save/load semantics. The code uses the GCS client, introduces caching toggles and a listing cap to avoid slow listings with millions of blobs, documents concurrency limits and failure modes, and shows the extension running in production with caveats. It also notes portability to S3/Azure and the ContentManager subclass used.
AI-written summary · about
chapters (10)
- 0:00Introduction to Jupyter and Google Cloud Storage
- 4:46Extending Jupyter with FS backend
- 9:05Designing GCS backend
- 15:39Implementing content manager
- 19:00Backend initialization
- 22:16Handling files and dirs
- 25:17Moving files between buckets
- 27:05Performance and caching
- 29:17Concurrency and limits
- 30:51Summary
related talks
28mBuild a data studio in your notebook with jupyter-fs
34mMoving notebooks into the cloud: challenges and lessons learned
32mSpin up Jupyter notebooks at scale and enhance your productivity
28mHosting Notebooks for 100,000 Users
32mHow to use fully managed Jupyter notebooks in Amazon SageMaker
25mOvercoming 5 Hurdles to Using Jupyter Notebooks for Data Science, by the JetBrai...
30mOnyxia: A User-Centric Interface for Data Scientists in the Cloud Age
40mJupyterLab: The Evolution of the Jupyter Notebook