Serverless Kafka Patterns.

on YouTube ↗47m

" summary

Practical patterns for marrying Kafka with AWS serverless: use API Gateway + Lambda to write to Kafka instead of always running containers; use RDS triggers or Lambda for CDC as an alternative to 24/7 Kafka Connect workers; run consumers as Lambda functions that call partner HTTP APIs; offload topics to S3 via Lambda for cheap archival. MSK Serverless fits short‑lived test clusters and feature branches. Start serverless for low traffic (examples as low as $5–$10/month) and move to provisioned capacity once throughput and predictability demand it.

AI-written summary · about

" chapters (15)

1. 0:00Introduction to serverless Kafka
2. 3:25Classical distributed streaming systems
3. 10:38Total cost of ownership in cloud
4. 17:00AWS serverless expertise tiers
5. 21:17When serverless doesn't fit
6. 25:11API to Kafka with serverless
7. 27:24Database changes to Kafka
8. 30:15Message broker to Kafka adapters
9. 32:20Kafka to external HTTP APIs
10. 34:06Archiving Kafka data to S3
11. 36:15Kafka Connect serverless patterns
12. 37:40Serverless Kafka clusters
13. 39:47Lessons learned from serverless
14. 43:21Measure before you optimize
15. 45:06Summary and Q&A

" related talks

• 49mServerless Architectural Patterns and Best PracticesSascha Möllering · GOTO Berlin 2018
• 51mServerless architectural patternsYan Cui · NDC Oslo 2018
• 12mServerless Stream Processing with Confluent Cloud, Kafka and AWS LambdaGeetha Anne · Moar Serverless 2022
• 48mServerless design patternsYan Cui · Voxxed Days Luxembourg 2018
• 55mServerless Design Patterns for the Enterprise with AWS LambdaTim Wagner · Philly ETE 2016
• 01h 02mServerless Architecture PatternsLynn Langit · NDC Minnesota 2019
• 40mCloud architecture patterns, how to implement them in a serverless wayJimmy Dahlqvist · Øredev 2023
• 28mBuilding Event Streaming Applications with Serverless KafkaRicardo Ferreira · Serverless Practitioners Summit 2019