High Performance Stream Processing
summary
Benchmarks show concrete tradeoffs: in-memory direct binding hits about 13 million 100‑byte events/sec on a single VM, RabbitMQ non‑persistent hits ~30k/sec (but batching to 10 → 158k, to 100 → 453k), and Kafka yields ~2 million/sec at 100 bytes and ~600k/sec at 1 KB. Tools used include iperf, dd, Kafka and Rabbit perf tests and custom test harnesses; disk, network and VM type matter. Serialization profiling finds Kryo roundtrip ~1.4µs (1,443 ns) and recommends custom Kryo serializers for hot POJOs. Reactive/ring‑buffer processing reduces allocations versus thread‑pool queues; JMX and config changes can silently cut throughput.
AI-written summary · about
chapters (20)
- 0:00Introduction to stream processing
- 2:19Benchmarking challenges and goals
- 5:51Hardware and network throughput measurement
- 12:56Disk performance and benchmarking tools
- 15:11Message size impact on throughput
- 17:00Kafka and RabbitMQ performance comparison
- 20:15Upgrades and configuration surprises
- 23:55Testing and verification best practices
- 27:57Network hops and module composition
- 32:45Serialization and deserialization costs
- 39:59Optimizing Kryo serialization
- 45:16Custom serializers in Spring XD
- 51:02Benchmarking serialization performance
- 59:53Processor speed and bottlenecks
- 1:04:09Concurrency and thread overhead
- 1:09:51Reactive programming with Reactor
- 1:13:21Ring buffer and event loops
- 1:16:14Scatter gather and parallel calls
- 1:20:09Micro batching optimization
- 1:25:08Optimization pitfalls and summary
related talks
38mStream-style messaging development
54mMicroservices with Kafka
42mIt's like 10,000 streams when what you need is a queue
24mComparing Streaming Data Options For Speed, Simplicity & Safety
47mBuilding High Performance Real-time Stream Processing Microservices
44mData Streaming: Which horse to bet on?
46mNative AMQP 1.0
41mAkka streams for high throughput data processing