Big Data with AWS: our lessons learned using managed services
summary
A practical account of a Big Data pipeline built on AWS using IoT, Lambda, Kinesis Streams and Firehose, EMR, Elasticsearch Service, S3, CloudWatch and CloudFormation to ingest vehicle telemetry, handle both streaming and batch processing, and store raw and transformed data. The implementation includes a demo that writes Kinesis records to S3 and reports operational lessons: massive simulated load (100 PB/day for 20M vehicles), message loss during a 500k-message stress test (~5,000 lost), limits like static Kinesis shards and no Elasticsearch autoscaling, high cost sensitivity (rough $260M/month estimate) and difficulties debugging Lambda.
AI-written summary · about
chapters (7)
- 0:00Introduction and context
- 8:10Managed services
- 10:38Mistake #1
- 14:15Mistake #2
- 16:41Mistake #3
- 24:31Demo
- 38:37Q&A
related talks
39mDeploying Batch and Streaming Architectures on AWS
38mServerless Data Streaming on AWS
47mBuilding Data Lake Platform Fully Serverless
47mBuilding Near Real-time Analytics Solution on AWS
37mIntelligent Serverless and Scalable Real-Time Data Pipeline using Kinesis, Fargate and CFN
32mBuilding and scaling a log analytics platform - a serverless approach
37mBuild a Modern Data Platform on AWS- 47mBuilding Data Lake Platform Fully Serverless