Conversational big data
summary
A Java-based messaging system hosts the client queue on the server, uses structural conflation to merge updates (add/subtract maps and counters) and throttles per-client delivery so 45,000 clients receive a fair stream instead of 100k noisy messages; single-hop latency hits ~8µs with larger messages showing GC variance, and conflation stabilizes client throughput under 10Gb saturation. It also implements windowed stream processing with an aggregate/compensate method to make sliding-window aggregates linear instead of exponential. An alpha client exists, integrates with Lager/React, and the approach demonstrably reduces bandwidth while preserving needed immediacy.
AI-written summary · about
chapters (24)
- 0:00Intro
- 1:30Infographic
- 2:38Last Mile
- 3:06Data Recovery
- 3:43Focus
- 4:51Firewall
- 5:10Message queues
- 6:05Cloud
- 6:36Twitter
- 7:57Network
- 8:42Relevant
- 11:09Exploit
- 11:43Behaviors
- 15:28Performance
- 16:54Client side view
- 18:37Latency profile
- 18:49Latency range
- 19:36Processing events
- 20:55Map operations
- 21:47Aggregate function
- 22:20Window function
- 23:30Sliding windows
- 24:56Compensate
- 26:41Periodic window
related talks
37mLet's build our own message streaming platform
50mPanel discussion: The new RabbitMQ Streams
41mTracing Real-Time Distributed Systems
01h 08mDistributed Java Databases Under the Hood
44mDeep Dive talk | Resilience in Microservices with Congestion Control
43mUnifying streaming and message queue with Apache Kafka
54mReacting to an Event-Driven World
26mBreaking Magical Barriers