Build a Real-time Stream Processing Pipeline with Apache Flink on AWS

on YouTube ↗44m

" summary

Demonstrates building a live AWS pipeline that connects Apache Flink, Amazon Kinesis, and EMR, with Elasticsearch for indexing and CloudWatch for metrics and monitoring. Covers cluster topology limits and single-master risk, using external checkpoints for state recovery, and shows autoscaling and failover handling for thousands of events (demo ~6,000), with concrete EMR provisioning commands and end-to-end operational automation.

AI-written summary · about

" chapters (8)

1. 0:00Introduction to stream processing
2. 3:29Why use Apache Flink
3. 12:14Setting up Flink on AWS
4. 20:01Running a Flink job
5. 23:45Monitoring with CloudWatch
6. 36:12Importance of monitoring
7. 39:02Handling failures in clusters
8. 40:58Q&A and summary

" related talks

• 30mBuild and run streaming applications with Apache Flink and Amazon Kinesis Data AnalyticsHausmann · Flink Forward Europe 2019
• 26mRunning Apache Flink on AWSRyan Nienhuis · Flink Forward Europe 2019
• 40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobsSteffen Hausmann, Danny Cranmer · Flink Forward Global 2021
• 23mCustomer Journey with streaming data on AWSRahul Pathak · Flink Forward Europe 2019
• 34mBeaming Flink to the Cloud @ NetflixMonal Daxini · Flink Forward Berlin 2016
• 20mGetting Started with Event Stream Processing via Apache Flink on Microsoft AzureIsrael Ekpo · ApacheCon@Home 2021
• 32mMove to managed analyticsDamon Cortesi · AWS AMER Summit May 2021
• 49mBuilding an Enterprise Data Lake Architecture on AWSStephan Schiller, Alena Schmickl · Porto Tech Hub Conference 2024