Developing Highly Scalable Image Storage Solution with AWS Serverless
summary
E-commerce image storage moves from ALB/EC2/EBS/EFS to a serverless stack using API Gateway, Lambda, S3, DynamoDB, SQS, SNS, EventBridge and Kinesis Data Firehose, handling millions of images daily. Key results: async thumbnail generation via S3 events + SQS, reference counting and cleanup driven by DynamoDB + scheduled EventBridge jobs, change log recorded with DynamoDB streams into a changelog table, cross-account CloudWatch logs piped to OpenSearch via Firehose, customer migrations in chunks (100 projects) with batch writes (up to 25 items) and on‑demand fallback.
AI-written summary · about
chapters (11)
- 0:00Intro
- 1:42Agenda
- 2:17Application architecture in 2017
- 4:34Application architecture in 2018 & 2019
- 5:47Motivation behind the reimplementation
- 10:02Current architecture
- 35:20Challenges & lessons learned
- 35:361. Architect with AWS serverless quotas in mind
- 44:052. General architectural decisions
- 46:223. Challenging to trace all requests
- 49:12Outro
related talks
46mHighly Scalable Image Storage with AWS Serverless
21mThe best compute for your storage - Amazon S3 & AWS Lambda
41mTechnology mix: Around the serverless in 180 days
39mServerless in production, an experience report
31mAccelerate business innovation with serverless applications
49mServerless Architectural Patterns and Best Practices
41mUnlocking Faster & Efficient Data Processing w/ Serverless
20mThe Serverless First Mindset