Orchestrating Data Workflows Using a Fully Serverless Architecture

on YouTube ↗26m

" summary

Describes Fundbox’s serverless data orchestration platform built with AWS Step Functions, Docker containers, Spark, and AWS Fargate to run hundreds of dependent data workflows. Covers using Step Functions state machines, the platform’s limitations, and custom scheduling and dependency features implemented to overcome those limits. Explains how Docker containers and Fargate were used to address resource bottlenecks in the serverless setup.

AI-written summary · about

" chapters (12)

1. 0:00Intro
2. 1:00Motivation
3. 4:24Guidelines
4. 7:55Detailed documentation and Examples
5. 9:40Monitoring jobs
6. 10:34Architecture - Monitoring
7. 11:40Job deployment
8. 14:08Job dependency service
9. 15:27Architecture. Dependency Service
10. 17:31Custom runtime support through Docker
11. 20:25Runtime Parameter
12. 23:17Job on-demand execution

" related talks

• 41mUnlocking Faster & Efficient Data Processing w/ ServerlessUma Ramadoss, Adam Wagner · GOTO EDA Day Nashville 2023
• 43mBringing EDA to the Containers WorldJessica Deen · GOTO EDA Day Nashville 2023
• 01h 18mBuilding applications with Serverless Framework ...Fredrik Vraalsen · Berlin Buzzwords 2019
• 20mModern Application Architecture with AWS Step FunctionsOskar Neumann · Conf42 Cloud Native 2023
• 29mOrchestrating complex (not complicated) tasks using AWS serverless and PythonMichael Kelly · PyCon AU 2019
• 34mDesigning Data Intensive Applications In Serverless ArchitectureNikolay Matvienko · Heapcon 2019
• 23mOrchestration Lightning: Rapid-Fire AWS Step Functions PatternsEric Johnson · GOTO Serverless Bengaluru 2025
• 51mChoreography vs Orchestration in serverless microservicesGuillaume Laforge, Mete Atamel · JCON-ONLINE 2021