Testing Big Data Applications (Spark, Dask, and Ray)
summary
Compares Spark Dask and Ray testability, shows a reusable pattern: separate core computing logic from orchestration and cluster setup, unit-test pandas or pure‑Python cores, run small integration tests on two months / two locations before full scale, and add data validation as acceptance checks. Uses NYC taxi data (parquet files 50–150 MB, 242 locations) and claims converting frames to 2D Python arrays can be 50x faster. Demonstrates Fugue decouples logic and execution so end‑to‑end tests run locally and scale to real backends.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:54Framework vs business logic
- 3:01Poor Spark function example
- 6:13Minimizing framework dependency
- 8:43Testing Dask and Ray logic
- 10:13Python iterables alternative
- 13:05Testing workflows best practices
- 20:28Fugue testing introduction
- 25:10Testing Fugue SQL logic
- 29:45Test types and design strategy
- 30:31Q&A
related talks
39mTesting Big Data Applications (Spark, Dask, and Ray)
46mSimplifying Testing of Spark Applications
39mUnit Testing for Data Scientists
39m"Unit tests for Data" with Deequ
53mTips and tricks for robust big data applications
25mTesting Data Pipelines
40mPerformance Testing and Big Data
28mHow to not pull your hair out while providing data to the business