Increase productivity and simplify data operations in Python
summary
Claims a concrete workflow: use Ibis to write one set of Python expressions, serialize via Substrait, and export models with ONNX so the same featurization runs on pandas, DuckDB, Snowflake or PySpark. Example code cleans and yields ~13.5 million records unchanged across backends. The talk shows the pipeline built end-to-end and reports that it eliminates the usual five‑week rewrite bottleneck between prototype and production, making deployment repeatable and verifiable.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:46Data scientist’s nightmare
- 5:38Flaws in data workflow
- 11:37Introducing Ibis
- 14:27Demo: Ibis with pandas and DuckDB
- 17:37Cross-backend data processing
- 20:29Substrate as universal language
- 23:20Prototype to production
related talks
26mUnified dataframe API for different data backend
29mData Science Best Practices : From Proof of Concepts to Production
01h 09mNotebook to Pipeline: Hands-On Data Engineering w Python
44mShaping the Future of Real-time Data Pipeline
17mProductionizing Data Science Workloads
38mBuilding Data Pipelines in Python
33mDealing with the versioning of production-ready models
27mNotebook To Production