Following the Data, Not the Function: Rethinking Function Orchestration in Serverless
summary
Pheromone implements a data-bucket abstraction and trigger primitives on Kubernetes (coordinator + workers) so intermediate objects themselves drive invocation instead of following a DAG of functions. It uses local shared-memory object stores and direct transfers, runs MapReduce without manual shuffles, and exposes a trigger API for custom patterns. Measured results: 40 µs local invoke (≈10× faster than CloudBurst), MapReduce app shrinks from ~6,000 to ~500 LOC, and end-to-end latency improves up to 1.6× on 10 GB shuffles. Open-source release.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:48Today's function orchestration
- 1:42Limited expressiveness
- 2:34Limited usability
- 3:56Limited applicability
- 4:53Desired function orchestration
- 5:56Key insight
- 7:02Data-centric orchestration
- 9:43Pheromone system realization
- 12:43Experimental settings
- 13:02Function interaction latency
- 13:47Case study: MapReduce sort
- 15:46Summary
related talks
16mBurst Computing: Quick, Sudden, Massively Parallel Processing on Serverless...
34mKubeFlow’s Serverless Component: 10x Faster, a 1/10 of the Effort
48mServerless beyond functions
22mLeveraging Service Meshes for Accelerating Serverless Workflows
38mBackground Processing, Serverless Style
21mAI Training Orchestration Evolution with Serverless Building Blocks
18mServerless Best Practices
52mServerless landscape beyond functions