Observing all the Serverless Things

on YouTube ↗36m

" summary

A serverless risk-compute system runs on Lambda, Fargate and S3 and needs baked-in observability; a 100+ resource demo stack fails under stress because DynamoDB is provisioned with 2 write RCUs causing throttles, which triggers SDK retries and produces four Lambda timeouts against a 6s limit and 502s from API Gateway. Resolution: switch DynamoDB to on‑demand provisioning, set SDK max retries to 3 (≈350ms) and fix a test that caused validation exceptions. Observability is auto-generated by the open-source plugin SLIC Watch which injects CloudWatch dashboards and alarms for Lambda, API Gateway, DynamoDB, SQS, Kinesis, Step Functions and more.

AI-written summary · about

" chapters (33)

1. 0:00Intro
2. 3:35Serverless HPC FTW
3. 5:54A typical case study
4. 7:29Observability in the cloud
5. 8:14Can we test our ovservability?
6. 8:55Alerts flow
7. 9:33Making sense of alerts
8. 10:23Initial hypothesis
9. 11:32Gathering evidence
10. 12:22Checking timeouts
11. 13:55Inspecting DynamoDB capacity
12. 14:46Tracing errors
13. 15:47HTTP 502
14. 16:50Lambda CloudWatch logs
15. 17:40Conclusions
16. 19:51Before & after
17. 20:25What we have learned so far
18. 21:09AWS native observability = CloudWatch
19. 22:08Alternatives outside AWS
20. 23:21CloudWatch out of the box
21. 24:32Getting the best out of CloudWatch
22. 26:33Best practices
23. 27:00CloudFormation for CloudWatch alarms
24. 27:18Can we automate this?
25. 28:10Intro to SLIC Watch
26. 28:50How SLIC Watch works
27. 29:48Before & after SLIC Watch
28. 32:35Configuration
29. 33:32How to get started with SLIC Watch
30. 34:51What if I'm not using serverless framework?
31. 35:18Roadmap
32. 35:33Wrap-up
33. 36:10Outro

" related talks

• 39mBuilding Resilient Serverless ApplicationsYan Cui · Reliable Web Summit 2021
• 39mObservability in DevOpsDipro Chatterjee · Geekle Free Online DevOps+JS Conference 2020
• 53mBottlenecks in Serverless ApplicationsHen Peretz · Serverless Architecture Conference 2019
• 28mCostly mistakes in serverless computingMiguel A. Calles · OWASP 20th Anniversary 2021
• 44mServerless Architecture on Production: Good, Bad and Ugly PartsSerhat Can · DevConf 2019
• 25mHow to Fail at Serverless (without even trying)Kam Lasater · Devopsdays Boston 2022
• 29mBuild highly scalable serverless applications on AWSFrank Tan, Cheyne Wagner · AWS Summit ANZ 2023
• 17mHow do we use Serverless Architectures on AWS LambdaSerhat Can · Voxxed Days Microservices 2018