Sol: Fast Distributed Computation Over Slow Networks

on YouTube ↗19m

" summary

Sol is a federated execution engine that pushes tasks to site managers to hide hundreds-of-milliseconds control latency and decouples communication from computation by splitting tasks into communication and computation subtasks. Prototype on EC2 across ten datacenters reclaims idle CPUs during remote reads, supports worker-initiated reschedule for mis-speculation, and reports average speedups of 16.4× on SQL workloads and 4.2× on ML jobs versus Apache Spark in resource-constrained networks.

AI-written summary · about

" chapters (5)

1. 0:00Start
2. 1:04Execution Engine: Core of Big Data Stack
3. 4:30Impact of Networks on Bandwidth-intensive Jobs
4. 9:42Challenge 1.2: How to Push Tasks w/ Dependencies?
5. 13:06Resource Decoupling in Data Plane

" related talks

• 21mMassive proofs for the massesByron Cook · O'Hearn Fest (POPL 2024)
• 30mFireside ChatAnatoly, Logan · Solana Breakpoint 2023
• 16mPreview: Cloud ComputingKshiteej Mahajan · OSDI '21 / USENIX ATC '21
• 22mProgramming the worldwide elastic supercomputer with UnisonPaul Chiusano · Scale By The Bay 2018
• 16mBurst Computing: Quick, Sudden, Massively Parallel Processing on Serverless...Daniel Barcelona-Pons, Aitor Arjona, Enrique Molina-Giménez, Stepan Klymonchuk, Pedro López-García · USENIX ATC '25
• 15mA General Framework for Communication-Efficient Distributed...Virginia Smith · MLconf SF 2016
• 25mElastic Scaling of Stateful Network FunctionsScott Shenker, Sylvia Ratnasamy, Justine Sherry, Shinae Woo, Sangjin Han, Sue Moon · NSDI '18
• 16mAutomatic Parallelization of Software Network FunctionsFrancisco Pereira, Fernando M. V. Ramos, Luis Pedrosa · 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24)