Building a Real-Time Analytics Database

on YouTube ↗47m

" summary

A practical recipe for a user-facing analytics DB: choose tuples (tables) stored columnar storage, ingest streams from Kafka, expose SQL, and implement multiple indexes (forward, inverted, Bloom, text, geospatial and a star-tree index) so high-cardinality, multi-dimensional queries hit 100ms-scale UI latencies rather than seconds. Apache Pinot-style architecture ships real-time ingestion, pluggable indexes, and distributed scaling; used at LinkedIn and UberEats for location-heavy, low-latency queries and it demonstrably delivers the freshness + concurrency tradeoffs required.

AI-written summary · about

" chapters (10)

1. 0:00Introduction and context
2. 3:31Defining real-time analytics
3. 6:19Data storage choices
4. 9:16Representing data models
5. 12:16The case for tables
6. 15:33Row vs. column storage
7. 18:25Query languages
8. 24:01Storage and immutability
9. 31:27Indexing in Apache Pinot
10. 41:58Performance and conclusion

" related talks

• 29mRealtime Analytics Unleashed: Unlocking the Potential of Data at ScaleBarkha Herman · Shift Miami 2023
• 46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and PinotK Wolok, T Berglund · Voxxed Days Zürich 2022
• 44mUSER-FACING, REALTIME ANALYTICS USING APACHE PINOTNeha Pawar · Crunch Conference 2021
• 34mReal-Time Analytics: Going Beyond Stream Processing with Apache PinotMark Needham, Karin Wolok · Open Source Summit Latin America 2022
• 38mReal-Time Analytics: Going Beyond Stream Processing with Apache PinotKarin Wolok, Rong Rong · Open AI + Data Forum @ Open Source Summit North America 2022
• 47mReal-time Analytics for Your Users with Apache PinotFrançois Teychene · Devoxx France 2022
• 47mReal-time analytics for your users with Apache PinotF. Teychene · DevOps D-Day 2022
• 33mOptimizing Speed and Scale of Real-Time Analytics Using Apache Pulsar and Apache P...Mary Grygleski · Reactive Summit 2022