Real-time stream data processing with fully managed AWS Kinesis services

on YouTube ↗32m

" summary

Telesign migrates SMS transaction and completion records from RabbitMQ into Kinesis Data Streams and runs hosted Apache Flink via Kinesis Data Analytics (Apache Flink) to join events, aggregate by country/operator with one‑minute tumbling windows, and persist results to RDS Postgres for live dashboards. The pipeline enforces exactly-once semantics and moves end-to-end latency from ~30–60 minutes down to 3–4 second latency, using CloudWatch for monitoring and a custom Kinesis publisher.

AI-written summary · about

" chapters (17)

1. 0:00Introduction and agenda
2. 3:24SMS Verify use case
3. 8:40Old architecture limitations
4. 10:20Goal of stream processing
5. 11:07What is stream processing
6. 13:08Batch vs stream processing
7. 13:52Stream processing architecture
8. 15:47Challenges in stream processing
9. 16:42AWS Kinesis services overview
10. 17:33Kinesis Data Streams explained
11. 19:40Kinesis Data Analytics
12. 21:12New architecture with Kinesis
13. 22:47Achieved latency improvements
14. 23:09Flink applications details
15. 24:30Apache Flink framework
16. 26:35Flink data flow and APIs
17. 29:15Q&A session

" related talks

• 30mBuild and run streaming applications with Apache Flink and Amazon Kinesis Data AnalyticsHausmann · Flink Forward Europe 2019
• 23mCustomer Journey with streaming data on AWSRahul Pathak · Flink Forward Europe 2019
• 35mStreaming and Real-time AnalyticsDonnie Prakoso · AWS Summit Online EMEA 2020
• 53mDefine data streaming architectures & solutions to fit your needsEd Berezitsky · AWS Summit Atlanta 2022
• 29mThe need for speed: Data streaming architecturesRajeev Chakrabarti · AWS AMER Summit May 2021
• 35mStreaming and Real-Time AnalyticsDonnie Prakoso · AWS Summit Online 2020
• 39mLarge scale streaming analyticsJay Palaniappan · Data Con LA 2019
• 55mCollecting Big Data at Scale with Amazon KinesisRohit Bhardwaj · Developer Summit 2020