Mainframe & Serverless Integration

on YouTube ↗47m

" summary

Describes a hybrid mainframe-to-cloud pipeline on AWS that streams hundreds of mainframe transactions per second into cloud systems using Kinesis and an S3 data lake, with indexing in Elasticsearch/OpenSearch and state in DynamoDB. The implementation uses dozens of Java AWS Lambda functions (with provisioned concurrency, retries and monitoring) plus Fargate containers and API Gateway to reconcile mainframe and cloud out-of-band, target sub-100ms user latency, and handle millions of transactions per day.

AI-written summary · about

" chapters (12)

1. 0:00Start
2. 8:54The Architecture (Extremely Simplified)
3. 10:20Quick Primer on Some Core AWS Services
4. 17:33Microservices Boundaries
5. 19:16Ensuring Correctness
6. 22:29Two Transaction Reconciliation Systems
7. 27:22Data Lake Integration
8. 29:26Circuit Breaker Pattern
9. 32:52Dead-Letter-Queue Pattern on Lambda
10. 34:45Testing & Debugging Serverless
11. 41:01Performance samples...
12. 41:46Takeaway System Optimizations

" related talks

• 37mIntelligent Serverless and Scalable Real-Time Data Pipeline using Kinesis, Fargate and CFNYogesh Sharma · DataEngBytes 2021
• 01h 02mServerless in production, an experience reportYan Cui, Domas Lasauskas · NDC London 2018
• 39mServerless in production, an experience reportYan Cui · Codemotion Milan 2017
• 31mAccelerate business innovation with serverless applicationsEmily Shea, Santiago Cardenas · AWS re:Invent 2019 Global Partner Summit
• 12mServerless Stream Processing with Confluent Cloud, Kafka and AWS LambdaGeetha Anne · Moar Serverless 2022
• 49mServerless Architectural Patterns and Best PracticesSascha Möllering · GOTO Berlin 2018
• 32mThe Serverless Tidal WaveArun Gupta · swampUP 2018
• 42mBuilding real-time serverless data applicationsAdam Wagner, Joseph Morais · AWS re:Invent 2022