Process Data Streams Efficiently in Elixir
summary
Claims that Elixir gains demand-driven, backpressured stream processing on the BEAM using GenStage and Flow, and shows a working pipeline that reads Twitter, partitions across five stages, uses windows every 100 elements and a flush threshold around 500 queued tweets, and computes top hashtags. Implementation uses a GenStage producer, a Flow analyzer with partitioning and reduce, a consumer, and a dynamic supervisor to spawn per-event workers with dispatch options like the partition dispatcher; the demo runs and returns sensible trending hashtags.
AI-written summary · about
chapters (6)
- 0:00Start
- 0:04What Our Data Streams
- 5:23Demand Driven Communication
- 8:48Computing a Word Count
- 25:31Analyzer
- 29:46Conclusion
related talks
25mGreen Benchmarking of BEAM Languages
25mMixing data streams at scale
08mReal-Time Data Systems: Speed, Scale, and Resilience
27mBatching Was Yesterday: Real-Time Tracking & Analysis For 100+ Million Visitors
41mElixir & Phoenix – fast, concurrent and explicit
41mCOVID-19 contact tracing on the BEAM
45mEverything Will Flow
57mWhy and how to transform a REST API into a Data Streaming API