PyData Stack: Pure Python open source data platforms
summary
Builds a pure‑Python data platform that actually runs locally and handles batch workloads up to tens of gigabytes using DLT, DuckDB, dbt, Dagster, Streamlit. Implements API pagination, file ingestion, and three schema‑evolution modes (evolve/freeze/discard) and wires DLT assets into Dagster jobs with dbt lineage and tests. Produces a Streamlit dashboard on Star Wars data that shows budgets jumping to ~\$400M on recent sequels and a surprising 70x ROI for A New Hope; the demo pipeline and tests run end‑to‑end and surface breakages for debugging.
AI-written summary · about
chapters (9)
- 0:00Introduction and context
- 3:09Data stack components
- 7:10Pipeline implementation
- 13:48Choosing data stores
- 15:54Data transformation strategies
- 23:00Orchestration with Dagster
- 28:55Data visualization
- 31:58Star Wars data analysis
- 35:12Q&A and closing
related talks
25mProcessing Biggish Data with DuckDB and Python
29mDagster: a Framework for Data Processing Applications
20mStreaming DataFrames: A New Way to Process Streaming Data
35mData Warehousing with Python
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
47mDagster: An open source Python library for building data applications
39mStreaming doesn’t have to be hard
46mFrictionless Data Science w/Python