Stream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way

on YouTube ↗27m

" summary

Live pipeline runs on a laptop: Kafka ingest -> RisingWave materialized views -> data persisted as Apache Iceberg with Polaris catalog on localstack S3, queried by SQL and shown in Streamlit. Demo builds a live leaderboard game, streams JSON events, creates tables and partitioned Parquet files, and surfaces real-time widgets and time-series metrics; snapshotting enables time travel and fast reads. Setup runs on Docker Desktop/Kubernetes and completes in minutes, not weeks.

AI-written summary · about

" chapters (8)

1. 0:00Introduction and motivation
2. 3:53Event-driven architecture basics
3. 10:12Streaming database and Iceberg
4. 15:25Demo setup overview
5. 20:03Creating tables and pipelines
6. 23:17Iceberg metadata and performance
7. 24:19Visualizing real-time analytics
8. 26:47Summary and resources

" related talks

• 49mStreaming VisualisationGuido Schmutz · Voxxed Days Zurich 2019
• 35mStreaming and Real-Time AnalyticsDonnie Prakoso · AWS Summit Online 2020
• 28mStreaming Lakehouse with Kafka, Flink and IcebergZabeer Farook · apidays Singapore 2025
• 28mApache NiFi, Kafka, RisingWave, Iceberg + Stock DataTimothy J Spann, Karin Wolok · Conf42 Python 2024
• 31mReal-time Analytics: Going Beyond Stream Processing With Apache PinotMark Needham · Big Data LDN 2022
• 35mStreaming and Real-time AnalyticsDonnie Prakoso · AWS Summit Online EMEA 2020
• 38mRealtime Data Activation - Analytics, Intelligence & Decision MakingMarcelo Wiermann · THE CONF 2017
• 34mReal-Time Analytics: Going Beyond Stream Processing with Apache PinotMark Needham, Karin Wolok · Open Source Summit Latin America 2022