S3 as the state store for stream processing systems
summary
RisingWave is a production stream processor that uses S3 as state store with a storage/compute separation to support large states without local disk limits, keep hot shards in EC2, and coordinate remote compaction to cut S3 access and CPU duplication. Recovery targets are seconds by replaying only an epoch ≈100 ms of updates; schema-change backfills scale out instantly (example: 2→6 nodes gives ~3× backfill speed). Latency and S3 request cost remain the main tradeoffs.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 1:48Batch vs stream processing
- 4:58Stream processing challenges
- 13:25State management issues
- 19:33S3 as state store solution
- 27:18S3 latency and cost trade-offs
- 31:17Benefits of S3 architecture
- 32:20Q&A session
related talks
40mThe Dark Secrets of Stream Processing
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
23mStream from multiple on-premises sources onto AWS
16mBreakdown data silos - Build a data lake on Amazon S3
21mOptimize and manage data on Amazon S3
29mEvent streaming with modern data pipelines in a SaaS architecture
45mData Stream Processing Concepts and Implementations
22mSiphon : Modern Data Stack with SF-CH & Iceberg