Optimize and manage data on Amazon S3
summary
Social media–scale case study: a customer ingests billions of small events daily into an S3 data lake (petabytes) using Kinesis Firehose, batches events to reduce request count, writes Parquet via EMR, and queries with Athena. Operations use S3 Storage Lens (29 metrics, daily snapshots, 15 months retention) plus Storage Class Analysis and S3 Inventory to drive lifecycle rules. Use S3 Intelligent-Tiering, multi-part uploads and byte-range fetches (8–16 MB parts), GUID prefixes to reach 3500 puts / 5500 gets per-prefix and automatic archival to Glacier (~$1/GB-year class cited).
AI-written summary · about
chapters (10)
- 0:00Introduction to S3 optimization
- 1:23Understanding application requirements
- 2:47Organizing data in S3
- 5:14Deep dive into S3 Storage Lens
- 7:20Continuous right sizing for cost
- 9:26Optimizing request costs and performance
- 10:50S3 infrastructure and performance
- 13:55Scaling request throughput
- 14:58Case study social media platform
- 19:15Key takeaways and summary
related talks
34mBest Practices for Implementing a Data Lake in Amazon S3
34mBest Practices for Implementing a Data Lake in Amazon S3
22mTurn Data Into Insights
43mManaging a Data Lake on AWS
44mOptimize and manage your data on Amazon S3
39mLarge scale streaming analytics
28mTurn Data Into Insights
30mRaw Data to Business Insights: What You Need to Build a Modern Data Lake