Batch Operations with Broadway
summary
Broadway pipeline ingests Stripe-like charge events and moves from per-message inserts to grouped writes, replacing single-row repo.insert with Ecto.insert_all after assembling maps in handle_message/3 and routing via put_batcher. handle_batch/4 receives batches and returns successes/failures. With a batcher pool and batch_size 500 and 1s timeout, observed throughput rises from roughly 800 messages/sec to about 750–1500 messages/sec in examples, and further improves when batcher concurrency is increased.
AI-written summary · about
chapters (6)
- 0:00Introduction to Broadway
- 2:32Data pipeline use case
- 4:15Implementing processors
- 8:21Using batchers
- 10:57Batching performance results
- 13:24Advanced tuning and conclusion
related talks
45mKEYNOTE: Announcing Broadway
47mA Batch of Commit Batching
45mBatch select, coroutines and a million RPS
38mHow we Migrated (...) Without Downtime using Elixir
27mPayment workflow with Ruby and Clickhouse
24mGoing CPU Bound
31mBatch Job / Queue Processing Without a Message Broker
20mBatching vs. Streaming - Scale & Process Millions of Measurements a Second