Ship Compute or Ship Data? Why Not Both?
summary
Kayak is a prototype that adaptively splits requests between a KV (ship data) path and an RPC (ship compute) path by tuning rate limit and request fraction with a decentralized dual control loop design, proving fast convergence and fairness across tenants. The system is self-regulated, implements both KV and RPC APIs on storage servers, and achieves sub-second convergence, improving throughput by 32.5%–63.4% on compute-heavy workloads and up to 12.2% on transactional or non-compute-intensive workloads.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:23Two Trends of System Design
- 1:25Ship Data OR Ship Compute
- 6:41Kayak Overview
- 7:27Algorithm Overview • Kayak proactively tunes
- 8:17Algorithm Design
- 8:52Dual Loop Control Our Solution RX dual loop control
- 9:44Evaluation Workloads
- 10:24Performance
- 10:59Convergence of Dual Loop Contra
- 11:30Faimess across Multiple Tenants
- 12:13Ship Compute AND Ship Data
related talks
44mUsing Mesosphere Enterprise DC/OS to Deliver Real-Time Microservices to a Global Fleet of Ships
12mHow the Data Center Shapes the Workload
45mRiak Pipe, Distributed Data Processing
43mSystems @Scale Summer Q&A Week 2
27mThe journey to ship enterprise grade, high transaction rate, resilient APIs
16mState-Compute Replication: Parallelizing High-Speed Stateful Packet Processing
30mA tale of two distributed systems through the lens of a developer
30mData Platform Infrastructure at Rivian: A Deep Dive