Optimizing Amazon EventBridge Pipes with Apache Kafka

on YouTube ↗37m

" summary

Confluent Kafka connects to AWS using EventBridge Pipes to route, filter and enrich Kafka records and keep consumer logic serverless; large records get uploaded to S3 and replaced with an S3 pointer using the claim-check pattern, preserving order and batches. EventBridge enforces a 256 KB limit (Kafka allows ~10 MB), so Pipes plus two small Lambdas handle S3 writes and lookups. API Destinations hit a hard 3,000 rps throttle. The pattern reduces custom consumers and works in production.

AI-written summary · about

" chapters (10)

1. 0:00Intro
2. 1:44The challenge
3. 3:57Why Kafka?
4. 9:12EventBridge with Kafka
5. 11:18EventBridge with Kafka alternative
6. 17:05How we did this
7. 23:26Implemented solution
8. 29:51AWS wishes
9. 34:49Summary
10. 36:25Outro

" related talks

• 38mEvent-driven ArchitectureDanilo Poccia · AWS Summit Online EMEA 2020
• 47mServerless Kafka Patterns.Taras Slipets · Devoxx Ukraine 2022: Secure our Future
• 42mBuilding real-time serverless data applicationsAdam Wagner, Joseph Morais · AWS re:Invent 2022
• 38mServerless Data Streaming on AWSAnahit Pogosova · Code Mesh LDN 2020
• 28mBuilding Event Streaming Applications with Serverless KafkaRicardo Ferreira · Serverless Practitioners Summit 2019
• 44mAdvanced Event-driven Architectures atSarah Hamilton · GOTO EDA Day 2022
• 56mDesigning Event-Driven Applications to Improve ScalabilityAna Barragán, Andrés Felipe Victoria · AWS Summit Bogotá 2022
• 12mServerless Stream Processing with Confluent Cloud, Kafka and AWS LambdaGeetha Anne · Moar Serverless 2022