Building a Real-time Analytics Application with Apache Pulsar and Apache Pinot
summary
Live pipeline ingests Wikimedia server‑sent events with the Pulsar Python client, publishes to a Pulsar topic, and lets an Apache Pinot real-time table consume that topic using a JSON decoder and field‑extraction transforms; a Streamlit front end queries Pinot every second and auto‑refreshes the dashboard. Demo shows continuous ingestion (the Pinot instance reaches 71 million docs) and supports high‑rate aggregates and distinct counts with low query latency, proving the Pulsar→Pinot→Streamlit stack handles fresh, high‑volume event analytics in real time.
AI-written summary · about
chapters (27)
- 0:00Introduction
- 0:35Presentation Introduction
- 3:57Types of Applications
- 5:10Examples
- 6:54Properties
- 7:53Apache Pulsar
- 10:09What is Pulsar
- 14:48What is Pinot
- 15:16Pinot Architecture
- 16:43Demo
- 17:07Architecture overview
- 17:37Wikimedia
- 18:04Events
- 18:17Live Demo
- 26:21Streaming API to Python Application
- 30:01Pinot Admin Add Table
- 31:21Streamliner
- 32:51Auto Refresh
- 33:21Querying
- 35:51Joining an aggregation
- 36:41Who is using Pulsar
- 37:01Summary
- 37:31Conclusion
- 38:01More about Pulsar
- 38:41Additional information
- 39:11Twitch Stream
- 41:51Superset
related talks
34mReal-Time Analytics: Going Beyond Stream Processing with Apache Pinot
47mReal-time analytics for your users with Apache Pinot
49mIntroduction to Real-Time Analytics with Apache Pinot
47mReal-time Analytics for Your Users with Apache Pinot
44mUSER-FACING, REALTIME ANALYTICS USING APACHE PINOT
46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and Pinot
39mFaster Analytics for Fast Data with Apache Pinot and Flink SQL
32mStreaming Analytics with Apache Pulsar