Powering OLAP at Uber using Apache Flink

on YouTube ↗15m

" summary

Apache Flink converts user SQL into production streaming jobs that read Kafka, apply transformations via AthenaX, and write Kafka output that Apache Pinot ingests for 1‑click ingestion; the stack runs across regions with bi‑directional Kafka replication and HDFS replay backfills (Cover+) to handle failures. Production footprint is 200+ tables with multiple TB of data, serving thousands of queries per second at a p99 100ms latency, demonstrating the pipeline’s scalability and operational reliability.

AI-written summary · about

" chapters (14)

1. 0:00Intro
2. 0:18About the Speakers
3. 1:24OLAP (OnLine Analytical Processing)
4. 1:50Use cases: Restaurant Manager
5. 2:10Use cases: Freight
6. 5:31Pinot @ Uber
7. 5:51Ingestion Challenges
8. 6:35AthenaX Example
9. 8:39Pre-processing vs on the fly
10. 10:09Backfill today
11. 11:25Backfill Proposal 1: Kappa+
12. 12:11Backfill Proposal 2 - Common Ingestion SQL
13. 12:52Active-Active: Using Kafka aggregation
14. 14:00Active-Active: Using HDFS aggregation for backfill

" related talks

• 39mFaster Analytics for Fast Data with Apache Pinot and Flink SQLChinmay Soman · Berlin Buzzwords 2021
• 46mOptimizing Speed and Scale of User-Facing Analytics w Kafka and PinotK Wolok, T Berglund · Voxxed Days Zürich 2022
• 44mUSER-FACING, REALTIME ANALYTICS USING APACHE PINOTNeha Pawar · Crunch Conference 2021
• 43mChange data capture in production with Apache FlinkDavid Morin, Yann Pauly · Flink Forward Europe 2019
• 49mIntroduction to Real-Time Analytics with Apache PinotTim Berglund · GOTO Chicago 2023
• 20mReal-time analytics with Flink and Druid #bbuzzJan Graßegger · Berlin Buzzwords 2016
• 32mKeynote: Apache Flink - Completing Cloudera’s End to End Streaming PlatformMarton Balassi, Joseph Witt · Flink Forward Virtual 2020
• 54mAn Introduction to Apache PinotTim Berglund · Devnexus 2022