Efficiently Executing R Dataframes on Flink
summary
A prototype embeds the fastR R implementation on the JVM using Truffle/Graal so R user-defined functions run inside Flink task managers instead of in an external R process. This removes inter-process serialization, allows R code to access Java/Flink objects directly, and only falls back to the external R interpreter for unsupported features. Benchmarks compare this approach to native Flink, Spark, and SparkR and show many UDFs reach near-native JVM performance while keeping the R dataframe API.
AI-written summary · about
chapters (7)
- 0:00Introduction to project
- 3:00Comparing Flink and native execution
- 6:26User-defined function execution
- 10:04HotSpot compiler basics
- 13:25Simplifying user-defined functions
- 17:14Performance and streaming
- 20:10Benchmark results and conclusion
related talks
47mScala: The Unpredicted Lingua Franca for Data Science
50mFast Analytics On Big Data
23mFuture: Simple Async, Parallel & Distributed Processing in R
29mMy Leap from R to F#
51mJava in production for Data Mining Research projects
43mInteroperability between R Language & other Languages
40m"Clojure in the Fintech Ecosystem"
49mThere is a missing tool in your Java data structure toolkit – DataFrames!