Kafka Deployment at Scale

on YouTube ↗35m

" summary

Criteo describes operating Kafka at massive scale: 10 million messages/sec (~500B/day) and ~180 TB/day compressed across ~14 clusters with 72‑hour retention and topics up to 1,300 partitions. The deployment uses a custom C# Kafka driver, a watermark injector for latency measurement, a custom Kafka Connect replicator between datacenters, separate front vs application clusters, and a slow‑broker blacklist; streaming processing is done with Spark and Flink while addressing partition imbalance and ordering/out‑of‑order join challenges during a protocol upgrade from 0.10 to 0.11.

AI-written summary · about

" chapters (11)

1. 0:00Criteo Kafka at scale
2. 3:38Kafka scale metrics
3. 9:19Custom Kafka drivers
4. 12:32Migrating Kafka protocols
5. 13:37Kafka latency SLAs
6. 15:03Watermarks for ordering
7. 20:28Kafka cluster isolation
8. 22:15Spark Streaming challenges
9. 27:21Streaming platform challenges
10. 30:36Kafka protocol updates
11. 32:30Q&A

" related talks

• 13mScaling Kafka on KubernetesShalini Katyayani Koney · Conf42 Kube Native 2025
• 46mKafka Streams at the Big Data Crossroads at Lightning SpeedFrançois Sarradin, Jérémy Sebayhi · Devoxx France 2022
• 39m400 Billion Messages a Day with KafkaMAULIN VASAVADA, Kevin Lu, Na Yang · Silicon Valley Code Camp 2018
• 53mDemystifying KafkaNakul Mishra · JDD 2019
• 32mScaling your Kafka streaming pipeline....Opher Dubrovsky, Ido Nadler · Scale By the Bay 2021
• 11mScaling Kafka in the CloudHeikki Nousiainen · All Things Open 2019
• 01h 00mKafka as a Platform: the Ecosystem from the Ground UpRobin Moffatt · NDC Oslo 2020
• 29mKafka on Kubernetes: Does it really have to be the hard way?Victor Gamov · Montréal Cloud Native Day 2019