Real time OLAP and Stream Processing Friends or Foes
summary
Claim: real-time analytics needs two different engine types because of a freshness vs latency vs cost tradeoff; practical demos use CDC plus SQL streaming databases such as RisingWave and stream engines like Flink, while OLAP systems like ClickHouse Druid and DuckDB favor column scans. Evidence: RisingWave reports 140+ GitHub contributors and 100k+ Kubernetes deployments; implementation details cover row vs column storage, materialized views, watermarking and failure recovery.
AI-written summary · about
chapters (13)
- 0:00Real-time OLAP intro
- 2:10Batch vs streaming history
- 6:30Why databases over engines
- 9:25Simplicity vs performance
- 12:38Streaming vs OLAP similarities
- 15:30Decoupled compute storage
- 18:26Application differences
- 21:19Optimization trade-offs
- 23:29Storage vs compute
- 26:19Query engines
- 27:47OLAP + streaming challenges
- 31:45Use cases
- 33:13Conclusion
related talks
32mBuilding Real-time Systems with Open Source Technologies
28mMoving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns
20mReal-time analytics with Flink and Druid #bbuzz
10mData Rivers — The New Analytics Architecture
13mBuild Real-Time Analytics Platforms for Financial Crime Detection
14mThe Streaming Plane
20mAnalyzing streaming data with Apache Druid
33mStreaming Databases