Practical Data Orchestration in Elixir
summary
Processes two billion records a day with a three‑engineer team by moving orchestration into Elixir, offloading heavy lifting to Postgres and small external tools (Python, DuckDB) and running tens of thousands of daily jobs from 130 customers. Uses Ecto + a stream-to-copy pattern to load CSVs, enforces uniqueness with a partial unique index, runs ~1,000 jobs/hour from a simple GenServer worker pool that scales via monitored BEAM processes, stores ~30M job rows, and plans a migration to Oban.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:59ETL system overview
- 5:46Diverse data sources
- 8:13Concurrent jobs with OTP
- 11:26Crash recovery
- 13:53Pain points and future
- 16:45General advice
- 19:11Closing
related talks
45mKEYNOTE: Announcing Broadway
47mData - The Land DevOps Forgot
32mRDBMS and Apache Geode Data Movement
12mReal Time Data Warehousing: A Journey from Batch to Streaming with Faust
36mSimplifying ETL with Clojure and Datomic
42mBuilding a Data Pipeline with Clojure and Kafka
45mRuby for the soul of Big Data nerds
33mAccidental DataOps: How we matured our organization with data, and how you can too!