High throughput data processing with GenStage

on YouTube ↗22m

" summary

Shows how GenStage composes producers, producer_consumers and consumers to implement micro‑batching, partitioning, retries and backpressure with concrete numbers and code. Demonstrates a basic producer/consumer with max_demand 100, a partitioned topology into 3 partitions plus a broadcast selector that routes events by color, and a custom rate limiter 10 every 5s that emits red events on a 5‑second window. Notes real use scaling to ~500 GB/day and points to Flow and Broadway as higher‑level libraries.

AI-written summary · about

" chapters (8)

1. 0:00Introduction and concepts
2. 2:25Partitioning data
3. 5:34Back pressure management
4. 6:47Error handling strategies
5. 7:43Basic GenStage example
6. 10:55Advanced topology example
7. 15:21Rate limiting demonstration
8. 17:53Ecosystem and Q&A

" related talks

• 45mKEYNOTE: Announcing BroadwayJosé Valim · Code BEAM SF 2019
• 48mTaming & Composing High Performance Stream-Oriented Processing SystemsRiccardo · Functional Conf 2017
• 31mHelp I need a stream processor - learning how to chose ...Andrew Psaltis · Berlin Buzzwords 2016
• 39mLessons From Our First Trillion Messages with FlowJohn Mertens · ElixirConf EU 2019
• 01h 00mIn Depth Kafka StreamsDaniel Hinojosa · Devnexus 2020
• 43mConnecting Reactive Applications with Fast Data Using Reactive StreamsLuc Bourlier · Scala Days Berlin 2016
• 44mImprove your App UX with one model processing insights from batch and streamsRobert Kubis · Voxxed Days Ticino 2017
• 44mData Processing With MicroservicesMichael Minella · Spring I/O 2017