Real-Time Analytics: Going Beyond Stream Processing with Apache Pinot
summary
Real-time analytics pipeline uses a Wikimedia server‑sent events feed, a Python producer, Apache Kafka for streaming, and Apache Pinot as a columnar OLAP store with schemas, time columns, transformations and star‑tree/inverted indexes. Demo shows streaming → Kafka → Pinot → Streamlit dashboard with per‑minute counts, unique users and domains. Claims: production Pinot deployments ingest around 1M events/sec and serve 250k qps with millisecond latency; demo forces segment rotation every 1,000 rows to show behavior.
AI-written summary · about
chapters (20)
- 0:00Start
- 1:02Dashboards / BI Tools
- 1:22User-Facing Analytics
- 1:40Machine Learning
- 2:07Why Should I Care?
- 4:01Examples of Real-Time Analytics
- 9:10Properties of Real-Time Analytics Systems
- 10:21Building a User-facing Real-Time Analytics System
- 11:27What is Apache Kafka?
- 12:25What is Apache Pinot?
- 13:40Apache Pinot Architecture
- 15:05Dashboard Architecture
- 15:54Our data set: Wikimedia Recent Changes Feed
- 26:51How Does Apache Pinot Store its Data?
- 27:12Segments
- 28:10Inverted Index
- 29:25Star-Tree Index
- 30:02Pinot Specific Optimizations
- 30:58Powered by Apache Pinot
- 32:07Takeaways
related talks
47mReal-time analytics for your users with Apache Pinot
49mIntroduction to Real-Time Analytics with Apache Pinot
46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and Pinot
43mBuilding a Real-time Analytics Application with Apache Pulsar and Apache Pinot
47mReal-time Analytics for Your Users with Apache Pinot
44mUSER-FACING, REALTIME ANALYTICS USING APACHE PINOT
39mFaster Analytics for Fast Data with Apache Pinot and Flink SQL
54mAn Introduction to Apache Pinot