The middle way for static typing in Spark DataFrames
summary
A new library Doric provides typed column expressions for Spark DataFrames, implemented with cats and shapeless and using sourcecode for error locations. It narrows runtime failures to reduces runtime errors to two kinds (missing column or type mismatch), validates column types at compile/run time, reports multiple column errors in one run, and links errors to code locations. Tested on Spark 2.4+ in a production filtering/modeling project, it is built, used, and shown to improve error diagnosis and readability while remaining compatible with standard DataFrame APIs.
AI-written summary · about
chapters (14)
- 0:00Intro
- 2:10Scala type system
- 3:11Spark DataFrame vs datasets
- 11:06Possible "errors" working in spark
- 17:47Benefits? Compile time errors
- 19:58Next? Runtime validation similar to Dataset
- 26:25Modularization FTW!!!
- 29:54What is doric??
- 31:56Doric is super easy to use
- 34:43Doric splits castings in two types of castings
- 39:09Doric syntax sugar level
- 43:11What about structs?
- 48:15Doric highlights
- 48:40Doric current state
related talks
16mFrameless: A More Well-Typed Interface for Spark
47mKeeping the fun in Apache Spark Datasets and FP
23mSpark Datasets: why they aren't great and what could be done
06mInterview
30mDataframely — A declarative, 🐻❄️-native data frame validation library
19mThe Power of Type Classes in Big Data ETL
14mMaking Data Quality an integral part of developing Machine Learning and Data Products
46mIntroduction to Apache Spark with Frameless