Serverless Data Streaming on AWS
summary
Pipeline ingests roughly 600k requests per minute (≈10k/s), about 0.5 TB/day of JSON and Parquet, using Kinesis + Firehose + Lambda + Kinesis Data Analytics to route events, convert JSON to columnar Parquet, and run online contextual bandit rewards for near‑real‑time personalization. Shard limits (1 MB/s, 1k records/s) force overprovisioning; batch writes experience partial failures so code inspects responses, uses exponential backoff with random jitter, and keeps Lambda concurrency under control to avoid a huge blast radius. The pipeline runs reliably in production.
AI-written summary · about
chapters (27)
- 0:00Introduction
- 0:23Agenda
- 0:36About me
- 1:32About Wild Arena
- 2:25Data
- 2:57Numbers
- 4:22Technical Details
- 4:50Two Parts
- 5:16Why Serverless
- 5:32Service Pipeline
- 6:21kinesis data streams
- 8:55kinesis API
- 11:01Lambda Functions
- 11:49Parallelization Factor
- 13:23Troubleshooting
- 16:03Floater
- 16:47Inspiros
- 18:04Streaming vs Firehose
- 19:13Architecture
- 19:23kinesis Data Analytics
- 22:19Wrap Up
- 22:30Kiwis Limits
- 23:32FireHost Limits
- 24:25Timeout
- 25:22Batch Operations
- 28:34Lessons Learned
- 29:32Questions
related talks
55mCollecting Big Data at Scale with Amazon Kinesis
39mLarge scale streaming analytics
35mStreaming and Real-time Analytics
35mStreaming and Real-Time Analytics
43mThe Art of Embracing Failures with Serverless Architectures
37mIntelligent Serverless and Scalable Real-Time Data Pipeline using Kinesis, Fargate and CFN
25mA simple serverless data pipeline
11mBig Data Analytics: Immutability & Idempotence