The Art of Embracing Failures with Serverless Architectures

on YouTube ↗43m

" summary

A serverless pipeline using Amazon Kinesis and AWS Lambda ingests over half a terabyte per day but loses data when SDK defaults and service limits collide: Kinesis shards 1MB / 1,000 recs/sec, putRecords batches up to 500 with partial failures, and Lambda concurrency default 1,000 (new accounts 100). Timeouts and retries amplify transient glitches into outages, a single poison‑pill record stalls a shard for 24+ hours, and exponential backoff with jitter plus explicit timeout and retry limits prove decisive to avoid cascading failures.

AI-written summary · about

" chapters (16)

1. 0:00Intro
2. 2:21What to expect
3. 2:52Beginning
4. 5:25Distributed systems
5. 6:31Cloud
6. 7:04Serverless
7. 8:47Distributed architectures
8. 12:10Kinesis data streams
9. 14:09Failures
10. 19:32Distributed systems
11. 20:56Failures
12. 30:26Lambda
13. 33:44Lambda: Failures
14. 39:35Lambda & Kinesis
15. 40:07Epilogue
16. 42:54Outro

" related talks

• 53mBottlenecks in Serverless ApplicationsHen Peretz · Serverless Architecture Conference 2019
• 25mHow to Fail at Serverless (without even trying)Kam Lasater · Devopsdays Boston 2022
• 38mServerless Data Streaming on AWSAnahit Pogosova · Code Mesh LDN 2020
• 51mServerless architectural patternsYan Cui · NDC Oslo 2018
• 39mBuilding Resilient Serverless ApplicationsYan Cui · Reliable Web Summit 2021
• 35mHow to Handle Errors and Retries in a Stateless EnvironmentNitzan Shapira · DevOpsDays Tel Aviv 2018
• 36mObserving all the Serverless ThingsPeter Elger, Guilherme Dalla Rosa · GOTO EDA Day 2022
• 34mDesigning Data Intensive Applications In Serverless ArchitectureNikolay Matvienko · Heapcon 2019