Kotlin for Apache Spark: Why the world needs one more API for Spark
summary
JetBrains’ Kotlin for Apache Spark library provides a concrete API that removes encoder boilerplate, implements typed Dataset support and uses Kotlin extension functions and null safety so a left join fails to compile instead of producing a hidden NullPointerException. It ships on GitHub as an official JetBrains project, offers helper DSLs like withSpark and withCached to manage session and caching, targets JVM performance comparable to Scala, and currently lacks ML bindings and polished UDF ergonomics so those are listed as work items.
AI-written summary · about
chapters (32)
- 0:00Introduction
- 0:57How did you fall in love with data
- 2:21What is Apache Spark
- 3:02My background
- 3:58My knowledge
- 5:45Supported languages
- 6:15Python
- 6:37Python code
- 8:36F literal
- 11:30JVM architecture
- 12:20Python Spark performance
- 14:41Color
- 15:53Type system
- 16:48JVM knowledge
- 18:11Color type system
- 18:30Spark example
- 19:25If expression
- 19:48Java
- 20:13Encoders
- 21:36Spark R
- 22:08Codeenemy
- 22:23Extension methods
- 22:44Join methods
- 23:16Null pointer exception
- 23:49Left join method
- 25:06Selfbuilding capabilities
- 26:17Use in Spark
- 27:19Create Spark session
- 28:37Spark session
- 28:58Benefits
- 30:20Improvements
- 30:44Github
