Data pipelines powered by Open source and fun
summary
Shows a five-step, pragmatic pipeline recipe (select sources, quality checks, business logic, orchestration, iterate) and demos it end-to-end with open source tooling: conda, intake, GitHub Actions and a Dagster-based UI. Builds a simple reader/cleaner that reads a few-MB stock CSV in ~1.5s and a larger pipeline that runs PCA, scaling and k‑means clustering, writes cluster outputs and logs timings; both pipelines execute successfully locally and expose per-op durations and logs for benchmarking.
AI-written summary · about
chapters (9)
- 0:00Introduction and fun element
- 2:30About the speaker
- 5:23Simplifying data pipelines
- 7:30Five simple steps to build pipelines
- 13:55Tools for the demo
- 22:09Demo setup and GitHub actions
- 25:45Live demo of simple pipeline
- 29:42Live demo of complex pipeline
- 32:12Takeaways and closing
related talks
43mData Pipelines à la Mode
23mA DIY Guide to Building Data Pipelines with Python
21mSoftware Best Practices for Data Pipelines: Form Energy's Multi-Day Storage R&D Pipeline 🌳
59mPolyglot Pipelines with Apache Nifi
24mUncovering Community and Project Insights Through Data Driven Methods
20mMaking Data Pipelines in R: A Story From A “Self-Taught” Perspective
39mData Pipeline Evolution
41mAccelerating privacy-enhancing data processing