Building the composable Python data stack with Kedro & Ibis
summary
Claim: Python can replace SQL for scalable transformations by using Ibis + Kedro to build a pipeline locally and flip a config to run on a cloud warehouse. The demo uses DuckDB, Postgres, CodeSpaces and shows an Ibis deferred dataframe API, a Kedro data catalog/nodes/pipelines graph, a trained classifier and runnable pipelines. It shows the backend switch works and notes 20 backends tested on every commit and reproducible runs with config-only deployment.
AI-written summary · about
chapters (14)
- 0:00Workshop overview
- 4:42Composable Python stacks matter
- 9:02Ibis and deferred API
- 17:53Transform flight data
- 23:05Solve flight data exercise
- 33:10Analyze flight delays
- 38:17Solve delay analysis
- 46:01Switch to Kedro pipelines
- 50:37Kedro data catalog
- 58:10Configure data sources
- 1:05:23Build Kedro pipeline
- 1:15:46Create Kedro project
- 1:20:20Run full pipeline
- 1:23:20Closing remarks
related talks
28mIbis framework, Making data science work at any scale
40mPyData Stack: Pure Python open source data platforms
25mProcessing Biggish Data with DuckDB and Python
39mPython and SQL: Better Together, Powered by DuckDB
37mSpark Backend for Ibis: Seamless Transition Between Pandas...
35mData Warehousing with Python
27mWrite portable CI/CD pipelines using Python
42mIbis: Don't Let the Engine Dictate the Interface