Simplifying Testing of Spark Applications
summary
Shows that using Fugue and native Python lets the same tokenization and sentiment code run on pandas or Spark without changing logic, avoiding local cluster spinups and costly row‑by‑row Python UDF serialization; demonstrates map_in_pandas and pandas_udf quirks and Apache Arrow speedups, uses deterministic sampling (0.01 of a 1.7M‑tweet corpus) to iterate locally, drives development with TDD to reach essentially 100% unit test coverage, and the demos and tests run end to end.
AI-written summary · about
chapters (21)
- 0:00Intro
- 1:00Why use Spark
- 1:31Python Spark
- 2:10pandas
- 3:11Python vs Spark
- 4:50Scenarios
- 9:01Python UDF
- 12:26Pandas UDF
- 13:43Pandas Type Hints
- 14:46Undecorated UDFs
- 15:23Functional API
- 15:56Spark DataFrame
- 17:22Databricks
- 20:47Kaggle Notebook
- 29:25Testing in Spark
- 31:08Exploration Goals
- 32:00Mindset
- 34:36Demo
- 35:07Notebook Environment
- 36:29Tokenization
- 38:31Sentiment Scores
related talks
39mTesting Big Data Applications (Spark, Dask, and Ray)
34mTesting Big Data Applications (Spark, Dask, and Ray)
39mUnit Testing for Data Scientists
18mSimple and Effective Spark Tests with Spark-Test
53mTips and tricks for robust big data applications
35mBreaking down large tests with a Saff Squeeze
39m"Unit tests for Data" with Deequ
01h 29mAdv. Software Testing for Data Scientists