Future: Friendly Parallel and Distributed Processing in R
summary
Claims a single, portable R API lets code be written once and run locally, on clusters or in the cloud using the CRAN packages future and future.apply (plus doFuture). Demonstrations include swapping base lapply for future_lapply to cut wall time from days to hours on a 12‑core machine, launching 36 Google Compute Engine VMs to run Monte Carlo jobs, and a batchtools backend for HPC schedulers. It auto-detects globals, relays stdout/stderr, records timing, and is used on hundreds of terabytes in production.
AI-written summary · about
chapters (24)
- 0:00Introduction and thanks
- 1:24Why use parallel processing
- 3:17Simple parallel example in R
- 5:23Parallel options in R
- 7:13Cross-platform parallel challenges
- 8:38Introducing the future package
- 11:08Concept of futures explained
- 13:22Hands-on with futures
- 16:57Future assignment operator
- 19:06Overhead and when to parallelize
- 20:54Future backends and portability
- 22:50Building blocks for parallelization
- 25:53Automatic dependency handling
- 26:54First proof of concept
- 28:03Future ecosystem growth
- 30:15Real-world genomics example
- 31:58Job schedulers and clusters
- 33:25Batch tools integration
- 34:28Cloud computing with futures
- 35:59Higher-level workflow managers
- 37:25Shiny and asynchronous processing
- 38:12Future features and improvements
- 41:02Progress tracking and hooks
- 42:25Summary and future directions
related talks
51mFuture: Friendly Parallel Processing in R for Everyone
01h 01mParallel Python Analyzing Large Data Sets
52mC++ Standard Parallelism
15mRay: A System for Distributed Applications
15mMaking Complex R Forecast App...
54mParallel Algorithms Ranges and oneDPL
42mRobust Automated Forecasting in Python and R
02h 50mFunctional Array Funhouse Intensive - Part 3 of 5