Data Pipelines à la Mode
summary
Describes building reproducible, incremental data pipelines by treating each step as a pure function that depends only on input snapshots; mutable sources are captured as S3 snapshots and pipeline results are cached by hashing the tuple of input-folder state plus the code image. Presents concrete implementations using Docker images on AWS (ECS/Fargate, Lambda, Step Functions), with example pipelines (word count and OpenStreetMap isochrones), and discusses Fargate startup/memory limits and a hashing-aware Airflow/Lambda executor prototype to avoid unnecessary reruns.
AI-written summary · about
