Using Elixir for scheduling cloudnative DAG workflows
summary
Handles a 53 TB dataset with 65 billion points and builds a distributed DAG runner in Elixir that boots per-job cloud instances to match resource needs, monitors GCP quotas, and runs any shell command as tasks. Uses an erl_exec wrapper instead of raw ports, isolates heavy jobs in Docker for end-to-end tests, syncs GenServer state to UI via a macro-backed LiveData hybrid with React, and ships a working prototype in two weeks as a few hundred lines.
AI-written summary · about
chapters (29)
- 0:00Intro
- 0:50Why big data
- 1:55Why build another workflow runner
- 2:31OLTP
- 3:40Python vs Elixir
- 4:57Cost effectiveness
- 7:58Proof of concept
- 10:14Monitoring tool
- 11:15Declarative state
- 12:15Data manipulation
- 13:25Update method
- 13:58Pull for changes
- 15:34Elixir macro
- 17:12Calling external processes
- 17:34Ports
- 18:54External Processing
- 21:17Web Interface
- 21:34Live View
- 22:49Hybrid approach
- 23:03Live Data
- 24:13Testing
- 24:57Integration tests
- 25:34What was needed
- 27:29Docker
- 28:42Server spawner
- 29:18OTP
- 30:24Questions
- 34:23Update Methods
- 34:57Live Data Case
related talks
37mRebuilding Workflow Orchestration in Elixir: The Story of Gust
03h 19mWorkshop - All Aboard The Elixir Express!
45mElixir: une potion pour mixer complexité et concurrence en temps réel
48mElixir by the Bellyful
47mOn Building Systems
53mElixir: a potion to mix complexity and concurrency in real‑time
53mIdioms for Building Distributed Fault-tolerant Applications with Elixir
53mWhat To Expect When You Are Elixiring