Complex Concurrency Patterns with Go
summary
Shopify’s Sarama is a Go Kafka client that processes hundreds of millions of messages per day and aims for high throughput and resilience. It uses per-partition goroutines, circuit breakers to fail fast on network calls, and a reference-counted broker pool guarded by a mutex to enable batching for throughput. The producer’s aggregator uses the nil channel trick to enable timed batches; retries were implemented with an infinite retry buffer (mistake) that later proved undesirable. The consumer mirrors partition ownership tokens and per-user response feeders with timeouts to isolate slow readers.
AI-written summary · about
chapters (13)
- 0:00Intro
- 2:41Literary Giants
- 14:25Dynamic Multiplexing - global, locked, reference-counted map
- 15:07Dynamic Multiplexing - acquire-broker
- 18:14Batching and I/O - aggregator
- 20:26Batching and I/O - buffer-producer
- 22:55Try, Try Again (option #1) for
- 23:27Try, Try Again (option #2)
- 27:59Putting it all together
- 32:57Structure Your Goroutines
- 38:05Ownership Semantics - broker
- 39:09Isolating I/O (redux)
- 41:46Feeding the User (response-feeder) for messages = range input
related talks
24mMastering Concurrency in Go: From Patterns to Production
05mGo for Real Time Streaming Architectures
29mGo Big with Apache Kafka
32mConcurrency Patterns in Go
28mManaging and Scaling Real Time Data Pipelines using Go
41mDetecting and Fixing Unbound Concurrency Problems
31mGo: Code that Grows with Grace
01h 01mRoundtable: Concurrency