Rebuilding Workflow Orchestration in Elixir: The Story of Gust
summary
Gust is a workflow engine implemented as Elixir macros plus a DSL, LiveView UI and a core app backed by Postgres and dynamic OTP supervisors; it replaces an Airflow stack that spawned 2 million work items by providing task management, scheduling, retries, observability, secrets and headless or multi‑node execution. Benchmarks show 4x less RAM at idle and 1.6x less RAM under load, producing roughly a 10x cost reduction. Gust runs legacy Python via Gust.By and preserves per‑run snapshots.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:24Speaker and Gust origin
- 2:09CompraCar workflow chaos
- 5:43Workers insufficient for orchestration
- 7:06What workflow orchestration means
- 8:33Understanding DAGs
- 9:38Code to executable workflows
- 10:20Real-time UI and observability
- 12:51Live demo workflow creation
- 14:38Airflow limitations and frustrations
- 19:50Coldplay scandal inspiration
- 20:55Why Elixir for Gust
- 21:37Gust architecture details
- 23:41Performance benchmarks and savings
- 24:44Gust expanded features
- 27:44Gust in Elixir ecosystem
- 28:46Best use cases for Gust
- 30:34From chaos to lightweight orchestration
- 31:15Q&A
related talks
37mUsing Elixir for scheduling cloudnative DAG workflows
41mIdioms for building fault-tolerant applications with Elixir
52mEvent Driven Workflows
16mExoFlow: A Universal Workflow System for Exactly-Once DAGs
44mGetting started with Orchestration and Choreography in Distributed Applications
49mThe Benefits of Orchestration in Distributed Systems
52mKeynote: Pilgrim's Progress to the Promised Land
23mDesigning a Workflow Engine From First Principles