Building Efficient and Frugal Image Data Processing Pipelines
summary
Pipeline recipes and measured results: switching PNG to WebP saves ~60% storage and moving 10 TB from S3 Standard to Glacier cuts a monthly bill from ~$300 to ~$30; processing 100k images over a weekend on AWS Lambda costs under $10 versus ~10x more on a single EC2. Use libjpeg-turbo, ImageMagick, Pillow, OpenCV for adaptive per-region compression (OpenCV: background q30 / faces q80), and orchestrate with Docker/Kubernetes, Ray/Spark/Dask and queues; cache transformed outputs and profile with AWS X‑Ray to avoid errors like reprocessing 3 TB of medical scans. switching PNG to WebP saved ~60% storage 100k images on Lambda <$10
AI-written summary · about
chapters (23)
- 0:00Efficient image pipelines intro
- 2:08Image processing use cases
- 8:53Pipeline success criteria
- 9:38Image formats choice
- 16:56Lossy vs lossless compression
- 19:24Compression tools/libraries
- 22:21Adaptive compression techniques
- 24:30Adaptive compression demo
- 25:56Storage cost strategies
- 28:50Serverless image compute
- 30:37Cloud image processing APIs
- 32:46Cloud cost optimization
- 34:31Data parallelism scaling
- 36:18Task parallelism workflows
- 38:29Distributed framework choice
- 41:01Containerization & orchestration
- 43:11Profiling & performance tuning
- 45:25Caching & redundant computation
- 47:54Pipeline observability
- 49:48ML-driven pipeline trends
- 53:28Ethics in image pipelines
- 56:16Summary
- 59:53Q&A
related talks
25mA simple serverless data pipeline
34mBuilding scalable end-to-end deep learning pipelines in the cloud
17mBoosting Cost Efficiency by Cost Aware Architecture
32mServerless Data Processing and Machine Learning
20mOpen-source processing pipelines for Cyclone Gabrielle
14mBattle-tested tactics to accelerate data science and ML R&D with AWS
43mData Pipelines à la Mode
27mVisualizing Cloud Systems