Testing Pandas- Shoots, leaves, and garbage!
summary
Shows concrete, runnable steps to make Pandas pipelines testable: a chained monthly_summary function on Alta ski data that shrinks ~14,000 rows to ~2,000 after grouping and exposes a bug (missing parse_dates) that tests catch and then fix. Uses pytest and ipytest for running tests, demonstrates Pandera + Hypothesis for generated-data property tests and Great Expectations for column-level assertions. Demonstrates pipe debugging and IPython set_trace and reports a passing test run in about 0.36s.
AI-written summary · about
chapters (7)
- 0:00Testing pandas
- 4:12Raw data to pandas chains
- 7:42Debugging with pipe
- 9:50Debugging in IPython/Jupyter
- 11:36Testing functions with pytest
- 15:04Advanced testing with Pandera/Great Expectations
- 26:25Q&A
related talks
21mHow to use Pandas Efficiently
25mTesting Data Pipelines
28mIdiomatic Pandas
01h 28mTesting with Pytest for Data Science
32mHow to do real TDD in data science? A journey from pandas to polars with pelage!
01h 26mPandas: .head() to .tail()
01h 21mPandas from the Inside
40mBeyond Pandas: The Great Python Dataframe Showdown