Real-time Analytics: Going Beyond Stream Processing With Apache Pinot
summary
Demo builds a realtime pipeline that pulls the Wikimedia RecentChanges SSE feed into Python, writes events to Apache Kafka, ingests them into a real‑time Apache Pinot table (with a Kafka connector, timestamp conversion and schema mapping), and surfaces aggregates in a refreshing Streamlit app. Claim: Pinot supports millisecond OLAP queries at thousands QPS; demo runs locally on a laptop with ~50–100 events/sec and an auto‑refreshing dashboard comparing last‑minute vs previous‑minute metrics. Also notes current version lacks joins and uses transformation functions for nested JSON.
AI-written summary · about
chapters (6)
- 0:00Introduction to real-time analytics
- 10:03Key properties of real-time systems
- 11:27Apache Kafka and Pinot overview
- 15:36Demo architecture walkthrough
- 23:40Building the dashboard with Streamlit
- 29:39Conclusion and resources
related talks
34mReal-Time Analytics: Going Beyond Stream Processing with Apache Pinot
47mReal-time analytics for your users with Apache Pinot
38mReal-Time Analytics: Going Beyond Stream Processing with Apache Pinot
49mIntroduction to Real-Time Analytics with Apache Pinot
44mUSER-FACING, REALTIME ANALYTICS USING APACHE PINOT
47mReal-time Analytics for Your Users with Apache Pinot
46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and Pinot
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale