Purrrfectly parallel, purrrfectly distributed
summary
purrr now maps in parallel using Mirai, an NNG‑based runtime that scales to millions of tasks over thousands of connections and claims up to a thousand times the efficiency versus naive approaches. Users start daemons (launch_local or launch_remote), SSH or SSH‑tunnel remote processes that dial back to a host URL, and assign named compute profiles to route GPU work to other machines; examples show mixing local and remote daemons (e.g., 6 + 6 = 12 workers) and working reliably in Shiny, plumber2 and tidymodels.
AI-written summary · about
chapters (6)
- 0:00Introduction to purrr
- 1:04Ecosystem and mirai origins
- 3:43Design philosophy of mirai
- 6:25Applications of mirai
- 8:34Distributed computing patterns
- 16:04Q&A and final remarks
related talks
20m{mirai} and {crew}: next-generation async to supercharge {promises}, Plumber, Shiny, and {targets}
23mA Haskell Auto-Parallelizer for Distributed Computing
01h 07mThe Asynchronous C++ Parallel Programming Model
16mTowards Optimal Rack-scale μs-level CPU Scheduling through In-Network Workload...
12mMonarch: An API To Your Supercomputer
20mAdvancing HPC The Hybrid Scheduler Approach at the D.E.Shaw Group
22mCustom Pipeline Parallelism for Frontier-Scale Model Serving
24mDistributed Black-Box Model Explanation with Ray