The Dark Secrets of Stream Processing
summary
RisingWave is a Postgres‑compatible distributed SQL streaming DB that adopts Backfilling Takes Forever, Consistency causes system stalls, and S3 costs spike with cache misses as concrete problems it confronts. The system uses S3 as primary storage and adds an EBS cache to cut S3 requests from ~2,000 to ~400 rps and drop a 12‑hour S3 bill from $20 to $3 in a benchmark; joins still stall under strong consistency and backfills can take arbitrarily long. Worth watching for hands‑on mitigation details and measured tradeoffs.
AI-written summary · about
chapters (12)
- 0:00Introduction to stream processing
- 1:45What is RisingWave
- 3:29Use cases in stream processing
- 5:37Design principles of RisingWave
- 13:25S3 as primary storage
- 19:27Challenges in stream processing
- 22:15Performance issues with S3
- 26:10Local SSD vs EBS caching
- 31:34EBS cache benefits
- 34:49Backfilling and scalability
- 36:31Consistency challenges
- 38:42Summary and lessons learned
related talks
40mS3 as the state store for stream processing systems
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
30mFive Challenges in Real-time Stream Processing and Five Solutions
45mData Stream Processing Concepts and Implementations
35mStreaming and Real-Time Analytics
28mBest of Stream + Batch: When Streaming Analytics Isn’t
39mAdvanced Stream Processing on the Edge
54mHow do you query a stream?