Unit Testing for Data Scientists
summary
The talk argues that unit tests give concrete confidence by isolating specific functionality and recommends starting small while refactoring. It demonstrates using pytest fixtures to share pandas DataFrames, creating a reusable local Spark session fixture scoped to the session so tests avoid spinning clusters, and using unittest.mock / magic mock with mock.patch to replace create_engine and pandas.read_sql so tests run offline and assert calls and return controlled DataFrames, which proves reliable for data workflows.
AI-written summary · about
chapters (19)
- 0:00Unit testing intro for data scientists
- 1:05Why unit testing matters
- 3:40Challenges adopting unit tests
- 5:27Data science workflow complexities
- 6:31Start small with unit testing
- 9:03Introducing pytest
- 10:08First unit test with pytest
- 12:20Pytest fixtures for test data
- 16:19Built-in pytest fixtures for debugging
- 18:07Advanced fixture features
- 22:24Mocking dependencies
- 23:30Mocking database access
- 26:39Understanding Python mocks
- 29:50Mock behaviors and return values
- 31:13Mock side effects and exceptions
- 34:05Asserting mock interactions
- 36:56Mocking pitfalls and gotchas
- 38:20Testing resources and reading
- 39:02Q&A
related talks
01h 29mAdv. Software Testing for Data Scientists
01h 28mTesting with Pytest for Data Science
39mTesting Big Data Applications (Spark, Dask, and Ray)
01h 30mAdvanced Software Testing for Data Scientists
34mTesting Big Data Applications (Spark, Dask, and Ray)
28mHow to not pull your hair out while providing data to the business
46mSimplifying Testing of Spark Applications
33mUnit Testing with Pytest