Leveraging GenStage to Implement Your Own Event Bus

on YouTube ↗25m

" summary

Frame.io replaces RabbitMQ/Kafka with a homegrown GenStage-based Event Bus that broadcasts standardized event structs to ~10–12 consumers, processes about 1.5M events/day (peaks ~2.2M), and keeps API p99 at ~101ms. Implementation uses a GenStage producer with broadcast dispatcher, experimented from fire-and-forget to Task.async_stream to GenStage to a consumer_supervisor spawning per-event workers (final benchmark ~300ms for mocked sleeps). System adds retries and failure logs but lacks durable persistence yet; webhooks were added by creating a new consumer.

AI-written summary · about

" chapters (24)

1. 0:00Intro
2. 1:47Agenda
3. 2:05Whats an Event Bus
4. 2:22Message Your Event
5. 2:46Event Bus Example
6. 3:46Event Bus Pros
7. 4:43Event Bus Cons
8. 5:20Code Snippet
9. 6:48Context Function
10. 7:31Notify
11. 7:54Event
12. 8:53Base Consumer
13. 9:18Requirements
14. 9:40First Example
15. 12:44What is GenStage
16. 13:11The Event Bus
17. 13:53Consumer Changes
18. 14:08GenStage Performance
19. 14:45BetterAST
20. 15:26ConsumerSupervisor
21. 15:41Demo
22. 17:43Event Bus Performance
23. 18:18Webhooks
24. 18:39Unit Testing

" related talks

• 45mKEYNOTE: Announcing BroadwayJosé Valim · Code BEAM SF 2019
• 57mGame of Streams: How to Tame and Get the Most from Your Messaging PlatformsMark Heckler · SpringOne 2020
• 40mTurn Hours into Seconds- Concurrent Event ProcessingEmil Soman · Functional Conf 2017
• 37mLet's build our own message streaming platformPiotr Gankiewicz · Rustikon 2025
• 54mGenStage and FlowJosé Valim · Lambda Days 2017
• 58mGame of StreamsMark Heckler · Devnexus 2020
• 55mKeynote: Announcing BroadwayJosé Valim · ElixirConf EU 2019
• 45mMicro Services and Events: Friends or Foes?Roland Tritsch · ElixirConf EU 2022