Scalable big data processing pipeline with Cadence
summary
A scalable big-data file-processing pipeline built using Cadence (Uber’s workflow engine) to handle large files, failures and peak loads. The talk describes using Cadence to coordinate tasks, enforce SLAs for processing time and quality, and to provide resilient, horizontally scalable resource management.
AI-written summary · about
chapters (11)
- 0:00Welcome
- 1:10Company and data
- 3:39Data processing challenges
- 5:48Pipeline solution search
- 9:52Choosing Cadence
- 11:15How Cadence works
- 13:23Workflows and Activities
- 15:31Solution implementation
- 17:42Scaling and performance
- 21:44Summary and recommendations
- 22:33Q&A
related talks
44mCadence: The New Open Source Project for Building Complex...
24mGotta Feed the Beast - Data Engineering at Scale
43mBuilding Scalable and Extendable Data Pipeline
45mResilient Predictive Data Pipelines
35mBuilding and Scaling Data Pipelines for Data Engineering With Ceph Object Storage
45mModern Data Architecture
30mDesigning Resilient Data Pipelines
16mData pipeline automation with built-in metadata and lineage (PRT054)