Real-Time Analytics: Going Beyond Stream Processing with Apache Pinot
summary
A working pipeline ingests the Wikimedia recent-changes stream into Python, writes to Kafka, and consumes into Apache Pinot with a Streamlit dashboard that auto-refreshes; the demo rotates segments every 1,000-row segment rotation so incoming events are queryable immediately. Claims: Pinot scales to >1M events/sec ingestion and serves 250k+ queries/sec at millisecond latency. Real examples: productized analytics at LinkedIn (≈800M users) and UberEats (≈500k restaurants).
AI-written summary · about
chapters (20)
- 0:00Start
- 1:02Dashboards / BI Tools
- 1:22User-Facing Analytics
- 1:40Machine Learning
- 2:07Why Should I Care?
- 4:01Examples of Real-Time Analytics
- 9:10Properties of Real-Time Analytics Systems
- 10:21Building a User-facing Real-Time Analytics System
- 11:27What is Apache Kafka?
- 12:25What is Apache Pinot?
- 13:40Apache Pinot Architecture
- 15:05Dashboard Architecture
- 15:54Our data set: Wikimedia Recent Changes Feed
- 26:51How Does Apache Pinot Store its Data?
- 27:12Segments
- 28:10Inverted Index
- 29:25Star-Tree Index
- 30:02Pinot Specific Optimizations
- 30:58Powered by Apache Pinot
- 32:07Takeaways
related talks
31mReal-time Analytics: Going Beyond Stream Processing With Apache Pinot
47mReal-time analytics for your users with Apache Pinot
49mIntroduction to Real-Time Analytics with Apache Pinot
47mReal-time Analytics for Your Users with Apache Pinot
38mReal-Time Analytics: Going Beyond Stream Processing with Apache Pinot
46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and Pinot
44mUSER-FACING, REALTIME ANALYTICS USING APACHE PINOT
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale