Build and run streaming applications with Apache Flink and Amazon Kinesis Data Analytics
summary
Builds a runnable Apache Flink pipeline deployed to Kinesis Data Analytics for Java Applications, ingesting via Kinesis Data Streams and writing to Amazon Elasticsearch/Kibana, replaying taxi trips at 25,000 events/sec. Starts with six shards, doubles shards to handle load, Flink scales by adding Kinesis Processing Units but triggers a savepoint and a minute‑or‑two downtime on resize. Monitors millisBehindLatest, incoming records/bytes and CloudWatch logs; warns against per‑event logging and embedding credentials.
AI-written summary · about
chapters (23)
- 0:00Introduction to session
- 0:26Streaming applications on AWS
- 1:31Taxi fleet use case
- 2:34Architecture overview
- 3:59Kinesis Data Streams setup
- 5:04Live architecture build
- 6:50KDA for Java application
- 10:53Configuring Flink job
- 13:02Deploying Flink application
- 14:06Ingesting taxi data
- 14:48Visualizing results
- 16:39Potential issues and scaling
- 18:26Scaling Kinesis stream
- 19:36Recap of architecture
- 20:40Best practices overview
- 21:02KDA for Java service
- 21:45Scaling Flink applications
- 23:18Scaling operations impact
- 24:43Key metrics to monitor
- 26:50Logging and debugging
- 28:16Security and data protection
- 29:41Further resources
- 30:22Q&A session
related talks
44mBuild a Real-time Stream Processing Pipeline with Apache Flink on AWS
40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobs
43mA Data Streaming Architecture with Apache Flink #bbuzz
19mBuilding Stateful Streaming Pipelines
40mLarge-scale near-real-time (NRT) data analytics platform empowered by Apache Flink
38mApache Flink: A Very Quick Guide
41mBeyond the Hype: When Apache Flink Solves Real Problems
32mReal-time stream data processing with fully managed AWS Kinesis services