Future: Friendly Parallel Processing in R for Everyone
summary
Introduces the future package as a minimal, cross‑platform futures API that actually runs on local multicore, SSH‑connected machines, HPC schedulers and cloud instances via backends like future.apply and future.batchtools. Shows concrete wins: chunking alignment jobs (hours per sample instead of days) and scalable Lapply replacement with preserved R semantics. Describes automatic global variable discovery via globals, backend testing, improved error/warning propagation, progress bar challenges, and sandboxed container execution for untrusted code.
AI-written summary · about
chapters (14)
- 0:00Introduction and motivation
- 2:21Why parallelize software
- 7:12Challenges in parallel processing
- 13:02Introducing the future package
- 17:16Futures in practice
- 21:31Future package features
- 27:09Future building blocks
- 33:01Real-world genomic example
- 38:31Cloud and HPC integration
- 39:42Future in other packages
- 42:27Testing and validation
- 43:52Advanced use cases
- 45:20Summary and call to action
- 46:25Q&A
related talks
44mFuture: Friendly Parallel and Distributed Processing in R
01h 01mParallel Python Analyzing Large Data Sets
52mC++ Standard Parallelism
01h 07mGeneric Parallel Programming
15mMaking Complex R Forecast App...
31mfunctime: a next generation ML forecasting library powered by Polars
54mParallel Algorithms Ranges and oneDPL
28mHow I used Polars to build built functime, a next gen ML forecasting library