Enriching your Data Stream Asynchronously with Apache Flink

on YouTube ↗26m

" summary

Shows how to replace blocking lookups with Flink’s async I O using RichAsyncFunction and demonstrates it running on Amazon Kinesis Data Analytics for Apache Flink. Demo calls an API Gateway + Lambda that sleeps 1.5s and proves async throughput scales as capacity goes from 1→2→4 doubling records/sec, while synchronous map stalls. Covers emitter thread, ordered vs unordered emit, timeout and capacity tuning, open/close for shared connections, RocksDB incremental checkpoints to S3 and CloudWatch metrics and autoscaling behavior.

AI-written summary · about

" chapters (7)

1. 0:00Introduction
2. 1:00Use Cases
3. 2:42Design Considerations
4. 5:33Synchronous vs Asynchronous
5. 6:38Why is Asynchronous Important
6. 8:15Asynchronous IO Architecture
7. 18:27Demo

" related talks

• 51mStream Processing Smackdown: Kafka Streams vs. Apache FlinkViktor Gamov · Voxxed Days Ticino 2025
• 42mStream Processing with Apache FlinkTugdual Grall · DevFest Toulouse 2017
• 39mEvaluating Streaming Framework Performance for a Large-Scale...Ron Crocker · Flink Forward Berlin 2016
• 41mStream Processor as a Database: Building Online ApplicationsStephan Ewen · Berlin Buzzwords 2016
• 40mThe Seven Deadly Sins of StreamingGiannis Polyzos · Big Data Conference Europe 2025
• 20mBatching vs. Streaming - Scale & Process Millions of Measurements a SecondAdam Butler · GOTO Chicago 2020
• 23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · Berlin Buzzwords 2021
• 40mReal-time Customer Engagement in Gaming Using Kafka and FlinkVolker Janz · GOTO Amsterdam 2023