Streams, External Services, and OTP
summary
Argues that describing external resources as Enumerable and Collectable lets core logic stay declarative and testable, and shows real code: a production pipeline pulling from Amazon SQS into a task scheduler. Reuses James Fish’s streamrunner and a custom stream router to add supervision, demand tracking and routing, but surfaces limits: composed streams block on long polling, aggregate-demand forces the slowest consumer to throttle everyone, and current API requires starting sinks before the source.
AI-written summary · about
chapters (24)
- 0:00Start
- 1:22Data is Messy
- 1:58Encapsulation
- 2:35Composition
- 3:30Supervision
- 3:59Enumerable
- 5:05Silly Example
- 6:44Now what?
- 7:03How about Task?
- 7:22No Mailbox Checking
- 8:05Stream Runner Courtesy of James Fish
- 8:36Suspended Streams
- 9:11The Loop
- 10:37StreamRunner Loop
- 11:11Demand/Supply Pattern
- 12:01Backpressure
- 13:59StreamRouter Loop
- 16:05Tracking Demand
- 16:27No Longer Good Enough
- 17:53Collectable Process
- 19:00Example Sink
- 19:44Supervised Stream
- 24:06Where are we?
- 25:11Remaining Questions
related talks
58mGame of Streams
59mGame of Streams: How to Tame & Get the Most from your Messaging Platforms
50mAddressing Recent Events
52mBeyond Microservices: Streams, State and Scalability
36mStream All The Things!
43mEvent Streams & Data Rapids: Harnessing the Full Power of Messaging Platforms.
47mStream All the Things!
42mIt's like 10,000 streams when what you need is a queue