Making Kafka Queryable with Apache Pinot

on YouTube ↗52m

" summary

Explains how Apache Pinot turns Kafka topics into a low-latency, SQL-queryable analytics store by ingesting Kafka directly into consuming segments on server nodes, writing immutable disk segments, and routing queries via brokers to servers; describes controller, minions and replication, and support for upserts. Shows the multi-stage vs single-stage engines, recommends low-level consumers for tight Kafka integration, and demonstrates the star-tree index that pre-aggregates dimensions to cut query time to ~50 ms at web scale with hundreds of servers and millions of segments.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 1:24Database history
3. 14:28Pinot architecture
4. 28:33Indexes in Pinot
5. 37:00StarTree and StarTree index
6. 41:57Data ingestion
7. 47:40Kafka integration
8. 51:36Q&A

" related talks

• 46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and PinotK Wolok, T Berglund · Voxxed Days Zürich 2022
• 54mAn Introduction to Apache PinotTim Berglund · Devnexus 2022
• 33mPinot, Why Are You So Fast?Tim Berglund · Scale By the Bay 2023
• 49mIntroduction to Real-Time Analytics with Apache PinotTim Berglund · GOTO Chicago 2023
• 44mUSER-FACING, REALTIME ANALYTICS USING APACHE PINOTNeha Pawar · Crunch Conference 2021
• 39mFaster Analytics for Fast Data with Apache Pinot and Flink SQLChinmay Soman · Berlin Buzzwords 2021
• 34mReal-Time Analytics: Going Beyond Stream Processing with Apache PinotMark Needham, Karin Wolok · Open Source Summit Latin America 2022
• 47mReal-time analytics for your users with Apache PinotF. Teychene · DevOps D-Day 2022