Augmented Data Processing at Massive Scale with Temporal
summary
Scale AI faces large intermediary payloads and brittle linear pipelines so it builds a generic run card activity that always reads from and writes to a centralized data set service and passes URI links between cards instead of chunky activity returns. Workflows become simple loops of cards, joins happen with DuckDB joins on links to produce new datasets, Python transforms run as spawned child workflows, and the UI drives execution interactively via Temporal signals to add and run cards step by step.
AI-written summary · about
chapters (10)
- 0:00Introduction and team
- 1:06Problem with agent evaluation
- 3:10Workflow complexity and patterns
- 5:38Temporal integration overview
- 6:19Building flexible workflows
- 8:23Run card activity design
- 10:28Architecture and data flow
- 12:38Inter-workflow joins and Python cards
- 13:42Interactive workflow building
- 15:49Q&A session
related talks
29mStructuring Data for Scalable AI Applications
58mAI-Enhanced Big Data: Integrating Private LLMs and Vector Databases
01h 29mFull-stack Machine Learning and Generative AI for Data Scientists
26mOne Pipeline to Rule Them All: Unifying Multimodal and AI Data Processing with Daft
46mPath of Data & AI Challenges
18mData Preprocessing for ML at Production Scale
27mAI-Powered ETL Pipeline Orchestration: Multi-Agent Systems in the Era of Generative AI
45mManaging Data & Analytics Projects Effectively