Building machine learning pipelines that scale
summary
A reproducible pipeline that predicts a chess live win probability gets built end to end using DT for ingestion, feature engineering in Ibis and Ibis ml, XGBoost for modeling, and Trino/Starburst Galaxy plus DataFusion (via let SQL) for scale and faster inference. The demo ingests ~30 GB locally then runs half a billion rows in the cloud, shows evaluation and mate features dominate importance, clock features matter, and model skill climbs with ply so the approach works but remains a simple baseline.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 7:52Data ingestion with DT
- 11:49Feature engineering with Ibis
- 58:52Building ML pipelines with Ibis ML
- 1:20:56Scaling on Starburst Galaxy
- 1:31:33Live demo: scaled inference
- 1:38:50Closing
related talks
18mAutomate the boring ML stuff with pipelines
20mDevMLOps at Scale
42mDeep learning beyond the learning
52mMachine Learning Data Pipelines
38mBuilding robust machine learning systems
38mReal Time ML Pipelines in Multi-Tenant Environments
27mHow to make ML teams 80% more productive.
18mBeyond Inference: Bringing ML into Production