Simple and Effective Spark Tests with Spark-Test
summary
A library named Spark-Test offers unit-test helpers for Spark 2.0–2.4 by providing a fast preconfigured local SparkSession and utilities to create DataFrames and Datasets. It adds an equals comparison that throws clear exceptions with detailed per-column and per-row diffs (and options to ignore columns), and notes planned support for unordered comparisons and RDDs; the implementation is available and prints actionable failure reports.
AI-written summary · about
chapters (7)
- 0:00Start
- 0:41Harrison Cheng
- 4:04Recipe for writing a unit test in Spark SQL
- 5:52Spark-testing-base and Spark-fast-test
- 10:02assertEquals - Example 1 with options
- 10:28assertEquals - Example 2
- 12:36Spark-Test - Upcoming developments
related talks
39mTesting Big Data Applications (Spark, Dask, and Ray)
46mSimplifying Testing of Spark Applications
39mUnit Testing for Data Scientists
34mTesting Big Data Applications (Spark, Dask, and Ray)
24mKONTEXTFREI: A NEW APPROACH TO TESTABLE SPARK APPLICATIONS
39m"Unit tests for Data" with Deequ
21mLarge-scale Data Quality Verification (...)
18mSpeedup unit testing with the Spock framework