Data Processing Pipelines for Small Big Data
summary
Claim: mid-sized datasets that fit on a laptop can be cheaper and faster than full big‑data stacks. Shows concrete techniques: generator-based ETL, async I/O / multiprocessing, SQLAlchemy bulk inserts, Numba JIT, Dask for parallel pandas, Parquet + AWS Athena, and Airflow on ECS with Celery. Reports a real win: reduced machine from 32GB to 8GB and processing time from one day to 30 minutes. Describes an opinionated feature store built on Parquet + Athena that works for offline use.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:08Small big data challenges
- 3:17Why small big data
- 4:41Talk objectives
- 6:26Python pipelines tricks
- 7:07EAFP performance trade-offs
- 9:13Iterators generators memory
- 12:01Async I/O threading processing
- 14:47Powerful ETL libraries
- 15:28SQLAlchemy ORM
- 19:20Numba JIT compiler
- 21:26Dask parallel processing
- 23:55Feature stores ML
- 26:22Lightweight feature store
- 29:54Airflow production pipelines
- 32:41Key takeaways and closing remarks
related talks
29mBuilding Large Scale ETL Pipelines with Dask
53mTips and tricks for robust big data applications
32mDataFrames: scaling up and out
47mHow to pick the best Data transformation strategy
25mProcessing Biggish Data with DuckDB and Python
45mAnalytics for not so big data with DuckDB
38mWeb Development To Big Data: A Journey
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion