Applying serverless architecture pattern to distributed data processing
summary
A container-based serverless pipeline that splits uploaded videos into per-second frame chunks, runs per-chunk OpenCV preprocessing and TensorFlow COCO object detection, then reassembles chunks with Pillow and writes results to S3-compatible buckets. Demo processes 24–120 frames per chunk, scales by starting parallel functions with ~300 ms cold start and a hot-worker mode for sustained work, and exposes GPU access as the main limitation. It works end-to-end but shows OpenCV thread-safety, frame-order/sync and audio-loss issues and nontrivial cost/network tradeoffs.
AI-written summary · about
chapters (6)
- 0:00Start
- 10:55Serverless data processing model
- 12:21From zero to serverless
- 13:03Native coroutine support
- 13:45Something special for Python developers!
- 30:24Key takeaways
related talks
41mUnlocking Faster & Efficient Data Processing w/ Serverless
21mLeveraging Python for Real-Time Image processing
34mBuilding scalable end-to-end deep learning pipelines in the cloud
26mServerless Architecture for Product Defect Detection
41mTechnology mix: Around the serverless in 180 days
19mHow to create a fast and cost effective serverless geospatial analysis pipeline...
49mBuilding Next-Generation Applications with EDA
32mServerless Data Processing and Machine Learning