Joining Dozens of Data Streams in Distributed Stream Processing Systems
summary
Claims that decoupling compute and storage by keeping state in S3-backed state lets a distributed SQL streaming database join dozens of streams without crashing, recover with near-zero downtime by booting new compute nodes that load state, and elastically scale by adding workers. Implements a custom RocksDB-like format for fast point reads, uses bushy join plans (not left-deep) and reports open-source release April last year, deployments in dozens of companies and hundreds of daily downloads.
AI-written summary · about
chapters (12)
- 0:00Introduction and background
- 2:10RisingWave overview
- 3:59Stream processing basics
- 5:45Why streaming joins matter
- 7:10State Management challenges
- 8:58Compute-storage coupling issues
- 11:30Cloud-native State Management
- 13:33Architecture comparison
- 16:24Failure recovery and elasticity
- 17:48Stream join algorithms
- 19:13Summary and conclusion
- 19:57Q&A session
related talks
32mBig Data Processing : Where do we go next ?
47mOne Does Not Simply Query a Stream
39mHow we isolate streaming ingest from search using RocksDB
39mHigh-volume Data Streaming and Storage
16mAStream: Ad-hoc Shared Stream Processing
55mDemystifying Stream Processing with Apache Kafka
41mStreaming: an Alternative to CRUD & Batch Processing
30mEvent Streaming and the Future of Applications