Future: Simple Async, Parallel & Distributed Processing in R
summary
Future provides a minimal, unifying framework for asynchronous, parallel and distributed R computing via the future package and adapters (future.apply, fur, doFuture), claims “write once run anywhere,” and reports production use since 2015 with 100+ immediate reverse-dependency packages exercised as tests. It now relays warnings, messages and output back in original order, adds a CRAN progress API progressor for cross-backend progress updates, and includes a future.test validation suite; it supports local, cluster and cloud backends and works in practice.
AI-written summary · about
chapters (10)
- 0:00Future: Simple Async, Parallel & Distributed Processing in R Why and What's New?
- 2:22Parallelization should be simple
- 6:08All we need is three building blocks
- 9:31User chooses how to parallelize sequential plan(sequential)
- 10:18Worry-free but does it work?
- 13:02Output and warnings behave consistently for all parallel backends
- 14:54progressr - Inclusive, Unifying API for Progress Updates Works anywhere - including futures, purrr, lapply, foreach, for/while loops....
- 15:50Developer focuses on providing updates Package code
- 16:40User decides how progress is presented # without progress updates
- 19:40Take home: future = worry-free parallelization • Developer what to parallelize c- User: how to parallelize • Stay with your favorite coding style • Automagic, e.g.globals, packages, output, warnings, errors, progress
related talks
29mMy Leap from R to F#
22mEfficiently Executing R Dataframes on Flink
22mUsing Python & R in Harmony
25mSql Server and R for Real-time Predictions
39mFuture of Forecasting the Future in...
43mInteroperability between R Language & other Languages
25mReproducibility Around a Web Application
15mStatically Analyzing the Dataflow of R Programs