Scaling Python-Powered Machine Learning With Snowflake
summary
Snowflake claims to run familiar pandas code at scale using Snowpark pandas, translating operations to SQL so 10 GB jobs run ~30× faster and a 1 TB job finishes under a minute in examples; feature engineering via Snowpark ML reports 25–50× speedups on 200M rows; distributed training uses a container runtime with Ray and prebuilt XGBoost/LightGBM libs; hyperparameter tuning and a partitioned workflow let teams build 60 million partitioned models in parallel, cutting an 18‑hour job to under 3 hours.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:04Snowflake platform benefits
- 2:53Pandas challenges for ML
- 4:40Snowpark Python API
- 7:08Pandas on Snowflake with Modin
- 9:55Feature engineering with ML Python
- 13:28Distributed model training
- 16:15Hyperparameter tuning at scale
- 18:24Millions of models with partitioned training
- 21:13Model inference and registry
- 23:00Batch and real-time inference
- 24:45Deploying with Streamlit
- 26:53Summary
related talks
23mAcceleratinag Data Science and Machine Learning with Snowflake DataTech
27mMachine Learning in the Warehouse with Python
21mMachine Learning in your Data Warehouse using Python
32mScalable Python and SQL Data Engineering without Migraines
01h 18mFrom raw data to interactive data app in an hour
13mLightning Talk: Highlights of the Snowflake Elastic Data Warehouse
21mUsing DBT and Python to Build Production Pipelines in Snowflake
36mScaling data workloads using the best of both worlds: pandas and Spark