Enterprise-Ready Python for High-Performance Data Teams
summary
pandas on Snowflake lets pandas-style code run at scale, claiming a terabyte workload in under a minute and up to 30× faster on a 10 GB benchmark versus vanilla pandas. The scaling layer comes from Modin translating pandas operations to pushdown execution. The managed container runtime loads 81 GB into a DataFrame in ~50 seconds (vs ~9.5 minutes), offers GPU-ready images, and yields distributed XGBoost training about 3–7× faster than the open‑source baseline.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 3:31Agenda and Snowflake overview
- 8:08Python workflow in Snowflake
- 9:11Pandas API capabilities
- 12:30Scaling pandas with Modin
- 17:07Python data ecosystem interoperability
- 18:31Snowflake Notebooks
- 22:41ML deployment challenges
- 24:25Snowflake ML for model development
- 27:12Container runtime for ML
- 31:02Automating notebooks and model refresh
- 31:47Streamlit dashboards
- 32:48Data to insights recap
- 33:52Key lessons and philosophy
- 35:34Q&A
related talks
32mDataFrames: scaling up and out
36mScaling data workloads using the best of both worlds: pandas and Spark
30mSpeed is Not All You Need for Data Processing
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
01h 09mNotebook to Pipeline: Hands-On Data Engineering w Python
31mComparing the Different Ways to Scale Python and Pandas Code
18mLooking under the hood - demystifying data tools
29mBuilding Large Scale ETL Pipelines with Dask