Large scale streaming analytics
summary
Shows an AWS serverless reference architecture that actually ingests, stores, analyzes and monitors streaming data: 100K records/second is the stated target and the presenter reports testing up to 1 million records per second (~180 TB/day) with end-to-end recommendation latency around 220 milliseconds. Uses Kinesis streams + Firehose to write raw data to S3, Kinesis Analytics or EMR/Flink for near‑real‑time aggregations, cross‑region replication to central lake, CI/CD via CloudFormation/terraform, CloudWatch monitoring and KMS encryption.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:43What is streaming data
- 4:33Build your own solutions
- 6:21Why cloudbased services
- 9:35Cloudbased services overview
- 11:06In ingest
- 14:00Transformation
- 17:35Kinesis Firehose
- 18:40Realtime analytics
- 21:26Streaming analytics
- 23:04Reference architecture
- 24:16Physical view
- 27:20CICD overview
- 29:23Monitoring
- 31:02Security
- 32:12Lockkey Framework
- 33:35Snowflake Architecture
related talks
35mStreaming and Real-Time Analytics
35mStreaming and Real-time Analytics
55mCollecting Big Data at Scale with Amazon Kinesis
38mServerless Data Streaming on AWS
23mCustomer Journey with streaming data on AWS
21mOptimize and manage data on Amazon S3
29mThe need for speed: Data streaming architectures
41mBuilding end to end streaming analytics serverless pipeline