Akka Streams for Large Scale Data Processing
summary
Credit Karma used Akka Actors and Akka Streams to build a high-throughput pipeline for tens of millions of users’ credit data: actors parallelize, parse and send records to an ingestion service, while Akka Streams pulls data through actors according to demand to control memory buffers and avoid OutOf-Memory errors. The talk covers applying parsing and simple business logic inside the stream and shares best practices for implementing Akka Streams at scale.
AI-written summary · about
chapters (9)
- 0:00Start
- 2:40Scale of
- 4:52Data Warehouse Import
- 9:56Warehouse Import Problems
- 11:29Akka Streams for fun and profit
- 18:42Warehouse Import Stream Code
- 20:57Analytics Export Stream Code
- 21:51Analytics Export leap Space Before
- 25:27Learnings
related talks
48mStreaming all the things with Akka Streams
31mSTREAM DRIVEN DEVELOPMENT - DESIGN YOUR DATA PIPELINE WITH AKKA STREAMS
25mSPARK AND AKKA FOR BIG DATA SYSTEMS, IN PRACTICE
46mTame Your Data with Akka Streams
33mKafka Microservices with Kafka plus Akka and Kafka Streams
01h 01mDrinking a river of IoT data with Akka NET
28mAkka - Enterprise Agentic AI with Tyler Jewel
26mHow to Pick Your Next Streaming Architecture