Building a Data Platform at Assembly Payments
summary
Assembly Payments builds a layered data lake using RDS and DynamoDB sources, AWS DMS/Firehose/Lambda for ingestion, Glue for landing-to-raw, DBT on Fargate (orchestrated by EventBridge and Step Functions) for raw-to-business, Snowflake for analytics and Looker for BI. The platform runs five data pipelines, ingests one million events, and supports 15 active users while processing 15 billion transactions lifetime. Serverless reduces ops and cost, Glue job bookmarks and TPU right-sizing matter, Step Functions work but get complex, dbt delivers tests and lineage, Snowflake gives fast reads but slow updates.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:30AGENDA
- 0:48WHO ARE WE?
- 1:12PLATFORM SNAPSHOT
- 1:24LEGACY
- 1:54ARCHITECTURE
- 3:30GLUE
- 4:38STEP FUNCTIONS
- 6:52SNOWFLAKE
- 7:37KEY TAKEAWAYS
related talks
39mCrafting modern data platforms: reveal your data value
49mBuilding an Enterprise Data Lake Architecture on AWS
43mManaging a Data Lake on AWS
42mBuild and automate a modern serverless data lake on
34mDesigning Data Intensive Applications In Serverless Architecture
25mA simple serverless data pipeline
29mHow financial services companies derive greater value from data with AWS
16mAutomated Data Analytics