Connecting Reactive Applications with Fast Data Using Reactive Streams
summary
Shows how to connect a reactive app to Spark Streaming using Reactive Streams and back pressure, with a Raspberry Pi demo that pushes up to 4,000 messages/sec and observes scheduling delay growing to seconds when overloaded. Spark 1.5+ reports processing rate and a dynamic PID rate limiter estimates safe intake each mini-batch; the demo’s connector implements Reactive Streams over TCP and forces the producer to slow to ~2,500/sec or drop low‑priority logs. Notes limitations: per-record time variance and Structured Streaming (Datasets) lacks back‑pressure support.
AI-written summary · about
chapters (6)
- 0:00Start
- 1:01Reactive applications
- 3:13Fast Data
- 9:51Spark Streaming
- 12:52Demo
- 23:24Back pressure in Spark 1.5
related talks
34mRetrofit Your Java App with a Reactive Flow Pipeline
38mRetrofit your Java App with a Reactive Flow Pipeline
01h 21mReactive Web Applications
28mBuilding a Realtime Receiver with Spark Streaming
48mTaming & Composing High Performance Stream-Oriented Processing Systems
30mFive Challenges in Real-time Stream Processing and Five Solutions
08mReal-Time Data Systems: Speed, Scale, and Resilience
27mRetrofit your Java App with a Reactive Flow