Faster Analytics for Fast Data with Apache Pinot and Flink SQL

on YouTube ↗39m

" summary

Combines Apache Pinot for low latency OLAP storage with Flink SQL for streaming joins and repartitioning, ingesting Twitch events via Kafka and decorating from S3 then visualizing in Superset. Shows Pinot indexes like inverted, JSON, geo and star tree and claims millisecond query latency at million+ QPS, uses Flink to emit key-partitioned upsert streams so group by stream id returns one, and notes an alternative segment-writer path gives micro-batch freshness tradeoff and cites Alibaba 2.5B events/sec as scale evidence.

AI-written summary · about

" chapters (10)

1. 0:00Introduction to real-time analytics
2. 1:14Real-time analytics use cases
3. 8:38Apache Pinot overview
4. 15:48Pinot ingestion challenges
5. 16:10Apache Flink introduction
6. 18:36Solving Pinot challenges with Flink
7. 25:54Live demo walkthrough
8. 26:35Key takeaways and summary
9. 27:18Pinot architecture deep dive
10. 28:43Q&A session

" related talks

• 46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and PinotK Wolok, T Berglund · Voxxed Days Zürich 2022
• 49mIntroduction to Real-Time Analytics with Apache PinotTim Berglund · GOTO Chicago 2023
• 34mReal-Time Analytics: Going Beyond Stream Processing with Apache PinotMark Needham, Karin Wolok · Open Source Summit Latin America 2022
• 34mNext generation OLAP stack using Apache PinotChinmay Soman · Berlin Buzzwords 2022
• 44mUSER-FACING, REALTIME ANALYTICS USING APACHE PINOTNeha Pawar · Crunch Conference 2021
• 47mReal-time analytics for your users with Apache PinotF. Teychene · DevOps D-Day 2022
• 54mAn Introduction to Apache PinotTim Berglund · Devnexus 2022
• 33mPinot, Why Are You So Fast?Tim Berglund · Scale By the Bay 2023