Zero-Administration Data Pipelines using AWS Simple Workflow
summary
Floto is an open source Python library (github.com/babbel/floto) that uses AWS Simple Workflow to run scalable pipelines with zero-administration and no master server or queue, claiming Amazon handles state so no cluster is required. It orchestrates deciders and activity workers (can be EC2 or Lambda), enforces idempotent and transactional steps, and guarantees tasks are not executed twice. A demo shows a decider surviving killed workers; the project is alpha but already running some production jobs at Babbel.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:08From scripts to tangled pipelines
- 4:29Spaghetti code pipelines
- 6:16Reliable idempotent pipelines
- 8:51Transactional failure-resistant pipelines
- 12:15Traceability and audit trails
- 14:28Scalability and chunking data
- 16:43Avoiding click-ops
- 18:15Failure and self-healing systems
- 20:47Understandable pipelines
- 22:38Alternatives to cluster management
- 24:03AWS Simple Workflow fits
- 25:35Live demo Simple Workflow
- 31:41Simple Workflow orchestration
- 33:05Comparison Airflow Lambda
- 36:21Visualizing Simple Workflow
- 38:05Q&A
related talks
44mCreating Python Data Pipelines in the Cloud
17mGoing with the flow? Apache Airflow for operational-quality scientific workflows
36mElegant data pipelining with Apache Airflow
26mOrchestrating Data Workflows Using a Fully Serverless Architecture
01h 00mDataflow Engines for Executing Cloud Applications: a Maslow's Hammer?
22mAirflow Beyond the Cloud: Python Workflows at the Edge
46mGetting Started With Data Pipelines Using Apache Airflow
25mBuilding advanced workflows with AWS Step Functions