Parallel computing with R using foreach, future, and other packages

on YouTube ↗20m

" summary

Explains how foreach’s %dopar% operator and a .combine like rbind let map/reduce loops run either sequentially or in parallel (demo uses two worker processes), how registerDoParallel(n) controls cores, and how doMPI/doRedis adapters support MPI and Redis clusters. Describes future (and doFuture) as syntax-agnostic parallelism that auto-exports lexical variables to workers and interoperates with foreach, notes a composition operator for nested parallelism, and mentions debugging options like stop-on-first-error and log capture.

AI-written summary · about

" chapters (7)

1. 0:00Start
2. 7:32Despite the for-loop syntax, foreach works kinda like
3. 10:13The future package is not opinionated, you can do the same thing with any syntax!
4. 10:40Scoped objects are auto-magically exported
5. 11:20Compose two foreach loops with the %% operator
6. 12:00Future works interoperably with foreach through doFuture, as do other parallel adapters
7. 14:41Don't mourn...

" related talks

• 51mProgramming in the TidyverseLionel Henry · satRdays Paris 2019
• 43mInteroperability between R Language & other LanguagesJohn D. Cook · GOTO Copenhagen 2016
• 46mElegant R: Tips from a Software DeveloperThales De Mello · The Conf 2018
• 22mUsing Python & R in HarmonyMatthew Brower, Krista Readout · PyOhio 2019
• 31mChasing PandasDaniel Baark · RubyConf MY 2017
• 28mRecords with Rank PolymorphismOlin Shivers, Justin Slepak, Panagiotis Manolios · ARRAY 2019
• 01h 31mFrom C to Comonads to Climate: A Functional Programmer's Journey in Array Programming(…)Dominic Orchard · FProPer 2024
• 28mData Science in Ruby? Is it possible? Is it Fast? Should we use it?Rodrigo Urubatan · The Conf 2019