Akka streams for high throughput data processing
summary
Credit Karma moves ~60M users worth of data and an actor-based pipeline hits limits when stages have different speeds, so the actor import and analytics exporters are rewritten as streams. Concrete wins: Akka Streams backpressure prevents unbounded queues, memory drops from ~28GB to ~~12GB, throughput rises to ~8–11M vs previous peaks, and a 10× spike (750k → 880k events/min) no longer halts the system. Benchmarks show actors 40m27s vs streams with mapAsync parallelism 28m (baseline 16m27s) and GC pauses drop seconds→milliseconds.
AI-written summary · about
chapters (20)
- 0:00Intro
- 3:49Scale of Data at Credit Karma
- 5:39Data Warehouse Import
- 10:50Warehouse import problems
- 12:03Akka Streams for Fun and Profit
- 15:44Akka Streams. Easy to Unit Test
- 18:05Akka Streams Built-in Stages
- 19:35Warehouse Import with Streams
- 20:50Warehouse Import Stream Code
- 22:29Analytics Export with Streams
- 22:46Analytics Export Stream Code
- 25:19Analytics Export Heap Space Pull
- 26:46Warehouse Import Garbage Collection
- 29:47Benchmarking Setup
- 30:43Benchmarking test Baseline Test, read parse XML
- 32:16Benchmarking Results
- 33:27Benchmarking Code - Akka Actors
- 34:14Benchmarking Code - Akka Streams
- 36:13Garbage Collection Times
- 37:35Learnings
related talks
27mUsing Akka Streams for Web scale Data Ingestion
48mTaming & Composing High Performance Stream-Oriented Processing Systems
28mBest of Stream + Batch: When Streaming Analytics Isn’t
01h 28mHigh Performance Stream Processing
39mEvaluating Streaming Framework Performance for a Large-Scale...
41mFast Data pipelines with Akka Streams and Alpakka...
44mData Streaming: Which horse to bet on?
11mBig Data Analytics: Immutability & Idempotence