KEYNOTE: Announcing Broadway
summary
Broadway is an Elixir library that wires GenStage pipelines into a ready data-ingestion framework: producers (SQS, RabbitMQ, GCP Pub/Sub) can run 4 producers → 20 processors with batch size 10, 1.5s timeout, automatic acknowledgments and graceful shutdown (drain/terminator). It implements back-pressure, partitioning and batching, reports minimal data loss, and ships metrics/telemetry plans; real users report GenStage handling bursts (Discord >1M pushes/min), AdRoll doubling throughput with Flow, and 1.3M push notifications in 10s.
AI-written summary · about
chapters (18)
- 0:00Intro
- 1:11Problem
- 2:24Broadway
- 6:34Requirements
- 8:55Collections: polymorphic
- 9:11Collections: extensible
- 9:40Collections: general
- 10:31Collections: tunable
- 11:24Collections: lazy
- 13:44Pipeline Parallelism
- 18:34GenStage: Demand-driven
- 25:19Dispatchers
- 25:56DemandDispatcher
- 26:44BroadcastDispatcher
- 27:07PartitionDispatcher
- 28:56Data ingestion / Data processing
- 36:29Complex GenStage pipelines
- 40:00Your turn
related talks
22mHigh throughput data processing with GenStage
25mLeveraging GenStage to Implement Your Own Event Bus
16mBatch Operations with Broadway
06mLightning talk - Broadway + Sqs + Apache Tika
20mPractical Data Orchestration in Elixir
39mLessons From Our First Trillion Messages with Flow
54mGenStage and Flow
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?