Scaling R with Spark
summary
Demo shows sparklyr adding real‑time streaming, pipeline export via MLeap, Kubernetes and RStudio Job Launcher integration, plus model extensions for xgboost on Spark. Local demo processes 10,000 rows in ~9 seconds and then—with Apache Arrow transfer enabled—processes 1,000,000 rows in about one second. Code examples use dplyr and DBI semantics, pipelines serialize to JVM runtimes, and models can be trained in‑cluster and exported for production.
AI-written summary · about
chapters (6)
- 0:00Intro
- 3:00R with Spark
- 4:40New in Spark
- 5:12Upcoming features
- 6:05Performance improvements
- 7:38Questions
related talks
42mScalable Data Science in Python and R on Apache Spark
16mPetabytes of data: How to use R at scale
37mBig Data Analytics, Machine Learning, Real Time Processing
43mDeploying Large Spark Models to production #bbuzz
52mSolving Analytical Problems using Apache Spark
39mAccelerating Data Science with RAPIDS
30mSimplify Big Data & AI on Spark and Ray with MLSQL
01h 25mApache Spark for Big Data Processing