Testing Big Data Applications (Spark, Dask, and Ray)
summary
Compares concrete testing techniques and a working tool: Spark Dask and Ray create brittle, cluster‑dependent tests while an abstraction — Fugue abstraction layer — lets the same code run locally or on a cluster so unit tests run on Pandas or Python. Shows a simple iterable implementation that outperforms Pandas by 50× faster. Recommends separating framework‑agnostic logic, running unit‑test on pandas, and adding data validation as acceptance tests to catch pipeline issues.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 2:12Testing purposes and types
- 4:38Designing testable apps
- 8:11Framework pitfalls
- 14:49Business over frameworks
- 16:59Common testing mistakes
- 19:52Scalable testing best practices
- 23:25Enforcing practices with Fugue
- 29:28Testing SQL and logic
- 33:28Summary
- 34:16Q&A
related talks
34mTesting Big Data Applications (Spark, Dask, and Ray)
46mSimplifying Testing of Spark Applications
39mUnit Testing for Data Scientists
25mTesting Data Pipelines
18mSimple and Effective Spark Tests with Spark-Test
40mPerformance Testing and Big Data
01h 29mAdv. Software Testing for Data Scientists
39m"Unit tests for Data" with Deequ