Streaming data analytics with Apache Flink
summary
Claims that a serverless Flink on Kinesis Data Analytics runs real-time pipelines at sub‑second latency, auto‑scales and provides multi‑AZ high availability; uses RocksDB state backend with async S3 snapshots and exposes Flink SQL via Kinesis Data Analytics Studio (Zeppelin). Demo ingests synthetic modem telemetry into Amazon MSK, runs tumbling‑window aggregations and pattern detection with Flink SQL, visualizes results, builds and deploys a continuous job that writes alerts to a Kafka topic for Lambda/SNS. Kinesis Data Streams offers 70 ms latency with enhanced fan‑out and up to 365‑day retention.
AI-written summary · about
chapters (12)
- 0:00Introduction and agenda
- 1:28Evolution of data processing
- 5:41Apache Flink overview
- 7:07Kinesis Data Analytics for Flink
- 8:53Kinesis Data Stream vs MSK
- 11:19Kinesis Data Analytics features
- 12:43Reference architecture pattern
- 13:46Demo setup and Kafka generator
- 15:32Flink SQL analysis in studio
- 17:37Real-time aggregation and visualization
- 20:45Deploying Flink application
- 22:09Recap and key takeaways
related talks
42mBuilding real-time serverless data applications
32mBuilding and scaling a log analytics platform - a serverless approach
35mHands-On Stateful Serverless Applications with K8s and Stateful Functions
12mServerless Stream Processing with Confluent Cloud, Kafka and AWS Lambda
41mBuilding end to end streaming analytics serverless pipeline
34mHands-on Stateful Serverless Applications with Kubernetes and Stateful Functions
47mBuilding Data Lake Platform Fully Serverless
38mServerless Data Streaming on AWS