Workshop 3: Data Engineering - A Roadmap... by Ramon Perez [Part 1]
summary
Roadmap claims a reproducible local-to-cloud data platform built with concrete tools: pandas, DuckDB, DVC plus Ibis, Typer and pathlib. It extracts bike‑sharing data (London, Seoul, Porto, Washington DC) from CSV/JSON/parquet/sqlite, builds an ETL/ELT pipeline, versions code and data, and produces a DuckDB local warehouse and a CLI that runs the workflow; the pipeline runs and yields a time‑series plot. Examples include a 1983–2020 wildfire dataset and a South Korea forecasting brief needing 24×365×2 = 17,520 hourly forecasts.
AI-written summary · about
chapters (10)
- 0:00Workshop introduction
- 3:09Motivation and roadmap
- 8:43Data engineering challenges
- 12:34Session tools overview
- 17:54Data pipelines: ETL vs ELT
- 25:51Reverse ETL use cases
- 29:26Simple pipeline demo
- 39:22Reusable scalable pipelines
- 42:34Extracting data sources
- 55:19CLI tools for pipelines
related talks
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
35mVisualizing research data: Challenges of combining different datasources
38mData Prep Kit: A Comprehensive Cloud-Native Toolkit for Scalable Da...
43mBehind the Scenes at Gapminder: A Story about Data and Visualizations
38mTransforming Data to Unlock Its Latent Value
24mLuigi & Data Pipelines
40mData-centric exploration using intake, dask, hvplot, and more
29m"Serpents and Ducks: wrangling data with Python and DuckDB"