GenStage by Example
summary
Claims GenStage turns an unbounded push pipeline into a demand-driven one with backpressure and shows it with Elixir code ingesting a CouchDB changes feed. The example builds a producer/broadcast and two slow consumers (each sleeping 2s) and measures a 154-event schedule and growing queues until GenStage enforces limits. Consumers request min 3 max 5 and start pulling at 50% capacity, producing a steady, predictable queue instead of memory blowup. Result: ingestion only happens when consumers can accept it, recovery needs tracking only a few events, and the example demonstrably works.
AI-written summary · about
chapters (8)
- 0:00Intro
- 1:50LET'S TALK ABOUT EVENT PROCESSING
- 4:29STREAMS
- 6:40SIMPLE PUBSUB
- 12:23DEMAND-DRIVEN
- 18:05EXAMPLE TIME
- 20:23PUBSUB IMPLEMENTATION
- 24:57GENSTAGE IMPLEMENTATION
related talks
58mGame of Streams
28mManaging and Scaling Real Time Data Pipelines using Go
27mStreams, External Services, and OTP
39mBuilding next-gen applications with event-driven architecture
19mBuilding a Distributed Data Ingestion System with RabbitMQ
27mScalable distributed systems from grounds up in Elixir
42mIt's like 10,000 streams when what you need is a queue
30mChange Data Streaming Patterns in Distributed Systems