DIY Async Message Pump: Lessons from the trenches
summary
Builds a real async message pump using TPL/async that runs until cancelled, shows SemaphoreSlim-based throttling to match a DB connection pool (example default pool ~100, demo limited to 3), and explains long polling vs SDK push. Compares transports: MSMQ, RabbitMQ, Azure Service Bus (atomic send/receive), Azure Storage Queues and SQS (visibility timeout), Kafka, AmazonSQS, SQL Server. Covers batching limits (256KB standard / 1MB premium for Azure SB, 64KB storage queues), visibility timeouts, and retry/transaction tradeoffs.
AI-written summary · about
chapters (18)
- 0:00Introduction to async message pump
- 1:04Patient record system challenges
- 3:12Message pump architecture design
- 6:21Basic message pump features
- 7:22Initial message pump implementation
- 11:14Rush hour and critical time
- 13:21Adding concurrency to pump
- 15:26Unbounded concurrency risks
- 18:38Limiting concurrency with semaphore
- 22:48Maintaining the abstraction
- 24:34Extending the message pump
- 27:18Transaction handling in pumps
- 30:07Cloud transaction modes
- 33:12Cloud latency and cost considerations
- 37:39Batching messages for efficiency
- 43:12Advanced pump requirements
- 44:55Monitoring distributed systems
- 45:37Conclusion and takeaways
related talks
01h 00mDIY Async Message Pump: Lessons from the trenches
56mDistributed Messaging patterns
01h 01mBeyond Pub/Sub - Advanced Messaging Patterns
01h 01mMessage Brokering and Event Driven Architecture on Azure
38mMessaging Patterns to Transform Your Cloud Architecture
50mMiddleware Showdown: Exploring Diverse Messaging Solutions
01h 03mChoosing the Right Messaging Solution for Your Architecture
39mDealing with eventual consistency