Keynote: Announcing Broadway
summary
Broadway is an open‑source Elixir library that builds concurrent, multi‑stage pipelines and claims to make ingestion from systems like Amazon SQS, RabbitMQ and Google Cloud PubSub simple and safe. It consumes from Amazon SQS, RabbitMQ, Google Cloud PubSub, wires producers and consumers with uses GenStage for demand-driven back-pressure, supports batching (example: batch size 10 or 1.5s timeout), partitioning, automatic acknowledgements, graceful shutdown with minimal data loss, and ships as v0.2 from Plataformatec.
AI-written summary · about
chapters (22)
- 0:00Intro
- 1:57Problem
- 2:54Broadway
- 7:32Requirements
- 9:47Collections: polymorphic
- 9:58Collections: extensible
- 10:15Collections: general
- 10:52Collections: tunable
- 11:06Collections: eager
- 11:57Collections: lazy
- 14:41Pipeline Parallelism
- 20:44GenStage: Demand-driven
- 27:31max_demand: 10, min_demand: 0
- 29:49Dispatchers
- 30:14DemandDispatcher
- 31:19BroadcastDispatcher
- 31:33PartitionDispatcher
- 32:00elixir Announcing GenStage
- 32:28Collections: concurrent (with Flow)
- 36:21Data ingestion / Data processing
- 43:49Complex GenStage pipelines
- 47:16Your turn
related talks
54mKeynote: Bringing a Ferrari to a Horse Race
52mKeynote: Pilgrim's Progress to the Promised Land
25mLeveraging GenStage to Implement Your Own Event Bus
45mKeynote: Architecting with Channels
45mMicro Services and Events: Friends or Foes?
27mKeynote II: Kafka and the Rise of Event-Driven Microservices
37mUsing Elixir for scheduling cloudnative DAG workflows
28mKeynote: The State of NATS as a Reactive Substrate