Using Akka Streams for Web scale Data Ingestion
summary
Iterable moves from RabbitMQ to Kafka to fix single‑point failures and spiky flow control, and wires Akka Streams + Alpakka Kafka to consume partitioned topics (partitioned by user ID) and keep per‑user ordering. The pipeline batches with groupWithin, uses mapAsync with parallelism=1 for ordered Elasticsearch bulk updates, merges per‑user updates, and uses Redis dedupe for at‑most‑once semantics. Result: steady throughput at tens of thousands/sec (peaks ~50k), far fewer outages, and correct ordering with retries for 429/409.
AI-written summary · about
chapters (25)
- 0:00Intro
- 1:25What is Iterable?
- 3:14About Iterable's Data
- 4:08Data inside ElasticSearch
- 4:20The Problem
- 5:01Existing Ingestion Architecture
- 5:32Motivation for change: RabbitMQ
- 6:49Motivation for change: Batching
- 7:31Example out of order processing
- 8:19Motivation for change: Error handling
- 9:11Spiky processing and non-optimal performance
- 10:15Real life effect at Iterable
- 11:01Goals of new architecture
- 11:58Tools and Technologies
- 14:11High Level Ingestion Architecture
- 15:50Kafka Partitioning strategy
- 16:37Code: Our streaming graph
- 16:47Source of Sources
- 17:24Source of messages per partition
- 18:14Batch messages
- 19:05Index Batch into ES
- 20:40Index batch into Elasticsearch
- 20:59Trigger downstream changes
- 22:03Commit back to Kafka
- 22:46Shutting down stream and cleaning up
related talks
41mAkka streams for high throughput data processing
01h 00mPutting Apache Kafka to Use
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...
42mBuilding a Data Pipeline with Clojure and Kafka
01h 00mIn Depth Kafka Streams
44mData Streaming: Which horse to bet on?
32mShould you read Kafka as a stream or in batch? Should you even care?
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka