Using ZMQ for Fun and Profit
summary
Shows how ZeroMQ composes a distributed pipeline with concrete patterns (Pub/Sub, Push/Pull, caution on Request/Reply), transports (inproc, IPC, TCP), and Python approaches from blocking sockets to zmq stream + tornado and an async loop. Recommends short stages, do serialization in workers, use namedtuples for multipart frames, and rely on HWM/backpressure. Describes an open source helper package (cs_erie) for channel config and reports a real-world swap from SQS to Kafka boosting throughput from ~3.5k to ~30k events/sec.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 1:40What is it
- 3:28ZMQ sockets
- 5:03Majordomo protocol
- 6:17PubSub
- 8:49Pipeline
- 11:15Lockstep
- 12:31Transports
- 14:29ZMQ Guide
- 17:43ZMQ Context
- 18:49Asynchronous Bits
- 21:21Socket Setup
- 26:28Scribe
- 27:32Subscription
- 28:10Multipart Messages
- 29:34Best Practices
- 29:53Logging Messages
- 30:06PiPI
- 30:31Jet Engine
- 31:55NameTuple
- 32:18DateTime
related talks
56mImplementing distributed applications using ZeroMQ, Python and other bad guys...
17mFun, interesting and common RabbitMQ use cases
47mAdopting Polyglot Flow
52mDIY Async Message Pump: Lessons from the trenches
57mGame of Streams: How to Tame and Get the Most from Your Messaging Platforms
50mPanel discussion: The new RabbitMQ Streams
37mLet's build our own message streaming platform
59mGame of Streams: How to Tame & Get the Most from your Messaging Platforms