Intake 2
summary
Intake 2 is a complete rewrite that replaces tangled legacy APIs with a simpler reader interface and removes unused features, yielding tiny reader definitions that auto-encode common loaders (pandas, dask, Ray, geopandas) and converters into plots or files. It builds declarative catalogs of datasets without storing data or running a server, supports pipelines as composable readers, and already guesses types and conversion paths. Current state is Alpha; many more readers, converters, and output nodes are planned.
AI-written summary · about
chapters (13)
- 0:00Intake 2 Alpha intro
- 1:29Intake in data landscape
- 3:17Missing piece in workflows
- 6:48Changes from Intake V1
- 7:10Data catalog programmatically
- 11:44Minimal dataset definitions
- 13:12Pipelines from readers
- 15:40Transforming data with converters
- 17:09Reading data with backends
- 19:39Intake 2 features summary
- 23:29Community involvement call
- 24:53Roadmap and next steps
- 25:34Q&A on writing and integrations
related talks
27mIngestify: Rethinking Ingestion for Complex Data
36mData pipelines engineering made simple with Scala
29mHow to make public data more accessible with "baked" data and DuckDB
05mLightning Talk: A Fast, Concurrent Data Loader for Time-Series Data
44mKeynote: Working Efficiently with Big Data in Text Formats
28mIbis framework, Making data science work at any scale
21mDesigning Modular Data Pipeline
30mPandas + Dask DataFrame 2.0 - Comparison to Spark, DuckDB and Polars