Harnessing Test-Driven Development & CI/CD for Smarter Data Analysis
summary
Builds a movie recommender from a 40,000‑row Kaggle set using TF‑IDF and cosine similarity, implements pytest unit tests with fixtures and parametrize (tests fail first then pass), extracts and modularizes cleaning functions, packages them with poetry into a wheel, and wires a GitHub Actions pipeline that builds tests and uploads artifacts to S3. Terraform creates an AWS Glue job (Python 3.9) that runs the prepared script and the demos and tests complete successfully.
AI-written summary · about
chapters (7)
- 0:00TDD and CI/CD for data analysis
- 2:54Why TDD matters now
- 15:36Modular data prep script
- 22:37Packaging utilities
- 26:13CI/CD pipeline with GitHub Actions
- 32:02Deploy to AWS Glue with Terraform
- 35:12Summary
related talks
01h 30mAdvanced Software Testing for Data Scientists
52mTest Driven Delivery of Infrastructure-As-Code
01h 29mAdv. Software Testing for Data Scientists
31mMaking AI work for you: Terraform engineer blueprint
27mFrom Training to Serving: Machine Learning Models with Terraform
28mHow to not pull your hair out while providing data to the business
21mSecuring your AWS Identity Management pipeline with PyTest
17mTDD for Infrastructure Code