Conversational Big Data
summary
A messaging system moves client-side queues to the server, measures per-device bandwidth and throttles or caches state, and avoids retransmit spam by sending only current deltas, yielding about 1.09 GB payload at 87.2% efficiency and up to ~5 million messages/sec in experiments. Benchmarks show Java ~5 µs and C ~2.4 µs. A small EP library (GitHub) implements replace conflation and reversible sliding windows to drop n^2 costs to linear, and the prototypes actually stabilize under load.
AI-written summary · about
chapters (31)
- 0:00Introduction
- 0:30What is big data
- 1:30Communication
- 2:04Data Distribution
- 2:30Home Broadband
- 3:10Machine to Human
- 3:35Traditional Messaging
- 3:48Message queues
- 4:18Reliability
- 5:33Cloud
- 6:12Big Data
- 6:35Human Experience
- 8:33Conversational Data
- 10:45Exploit Physical Data Structures
- 11:05Messaging
- 11:55Structural Conflation
- 12:22Deep Dive
- 15:42Regular Messaging
- 19:11Business Value Messaging
- 21:23CEP Simplified
- 23:03Window Behavior
- 24:13Sliding Windows
- 25:21Periodic Windows
- 25:58EEP
- 26:18Performance
- 27:48Sliding Window
- 27:58Compensate
- 28:40Performance Risk
- 28:58Reversible Computing
- 29:36Conversational Big Data
- 30:05Questions
related talks
37mUsing small data in the client instead of big data in the cloud
01h 07mBig Data in Memory
43mRealtime performance at scale (...)
33mReaching Ludicrous Speed
30mCaching, Payloads, and Other Dark Arts
29mHow We Reliably Send a Million Messages Per Request
10mConsiderations for a large-scale low-latency system
43mCaching and Beyond - Smarter Data Processing with Java