Kafka Streams at the Big Data Crossroads at Lightning Speed
summary
Carrefour describes migrating Spark/Hadoop batch jobs to Kafka Streams on GCP to run near-real-time pipelines for national store and warehouse stock and pricing. The architecture uses GCS, GKE, Bigtable, Kafka Connect, Schema Registry and RocksDB, with tests using embedded Kafka and Prometheus/Grafana for metrics. Operational challenges covered include slow state initialization (hours or days), topic compaction tradeoffs, consumer-group naming, lag and end-to-end metrics, and handling hundreds of millions of messages during backfills.
AI-written summary · about
chapters (10)
- 0:00Start
- 1:45Phenix: Carrefour's Data Lake
- 3:11What is Kafka Streams?
- 4:19Example of a Topology
- 5:47Kafka & Kafka Streams Ecosystem
- 8:33Teams Starting from... Scratch
- 10:15Changes to the Platform Over the Past 2 Years
- 18:35Solution 1: Blue-Green Deployment
- 22:22But What About Order in Joins?
- 28:36(8-5) A Few Specifics...
related talks
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...
35mKafka Deployment at Scale
01h 00mPutting Apache Kafka to Use
01h 07mKafka Streams and KSQLDB
01h 00mIn Depth Kafka Streams
39mData Streaming? I don't even know her
30mArchitecting Applications With Multiple Open-Source Big Data Technologies
50mApplication development and data in the emerging world ...