Unifying streaming and message queue with Apache Kafka

on YouTube ↗43m

" summary

Uber builds a message-queue layer on Apache Kafka that adds dead letter queues, ack/nack out-of-order with configurable retries and delay, and in-process fanout so consumers can parallelize without blocking Kafka commits. The system handles tens of petabytes and tens of thousands of topics, preserves offsets for cross-DC failover, replicates topics via an in-house replicator, and exposes simple language-agnostic handler APIs so Java/Python services can redeliver, purge, or requeue failed messages.

AI-written summary · about

" chapters (9)

1. 0:00Start
2. 1:30Agenda
3. 2:05Use Cases in Scale
4. 2:43Use Cases
5. 17:36Reliability
6. 18:07Cost Efficiency
7. 27:26Future Work
8. 41:02How Do We Deal with Duplicate Data on Kafka
9. 41:52End Users Deal with Duplicates

" related talks

• 48mKafka as a Message Queue: can you do it, and should you do it?Adam Warski · Devoxx Belgium 2017
• 30mEvent Streaming and the Future of ApplicationsBen Stopford · JAX 2019
• 50mIntroduction to Apache KafkaJames Ward · Devoxx US 2017
• 55mDemystifying Stream Processing with Apache KafkaEwen Cheslack-Postava · Philly ETE 2016
• 53mDemystifying KafkaNakul Mishra · JDD 2019
• 41mApache Kafka simply explainedOlena Kutsenko · Berlin Buzzwords 2022
• 45mAdvanced Kafka Features: From Dynamic Consumers to Fault ToleranceMaksim Sumin · JPoint 2023
• 01h 31mDeveloping Real-Time Data Pipelines with Apache KafkaJoe Stein · SpringOne2GX 2015