Turn Hours into Seconds- Concurrent Event Processing
summary
Case study refactors a slow, lock-based ranking job into a message-driven pipeline using Flow and GenStage, replacing global locks with a producer queue and actor-style stages. The scoring becomes idempotent scoring per player, partitioned so each player always hits the same consumer, enabling caching and eager loading. Benchmarks: a 100-game batch drops from ~48s to 3s, 400 games from 6m to 15s, and 2,500 games from 114m to 55s, proving the pipeline both correct and massively faster.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:32Presentation Overview
- 4:08Company Problems
- 5:45Beta Problems
- 8:14Actor Model
- 8:48Message Puzzle
- 10:04Processes and OTP
- 11:30Jen Stages
- 15:52Flow Functions
- 18:05Flow is Concurrent
- 22:42Ranking Players
- 25:00Ranking Pipeline
- 34:39Benchmark
- 35:56Conclusion
related talks
54mGenStage and Flow
25mLeveraging GenStage to Implement Your Own Event Bus
45mKEYNOTE: Announcing Broadway
24mGoing CPU Bound
01h 04mParallel Game Engine Design
49mBEAM Concurrency in Action
46mMulti-Priority Graph-Based Parallel Execution Using Taskflow
30mThe untapped power of parallel programming: build efficient code that scales