Patterns for Collaboration Between Data Scientists and Software Engineers
summary
Describes practical collaboration patterns between data scientists and software engineers using CI-driven model pipelines that run parameterized Jupyter notebooks on every commit to enable continuous evaluation and detect regressions. Shows a small Flask web UI plus an Airflow/Luigi pipeline that turns notebooks into reusable templated reports runnable with one click on replica datasets, and a documented data API to avoid manual exports, reducing stale analyses and producing reproducible reports for production.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:20Data scientists vs engineers
- 7:35Collaboration challenges
- 12:16Exploration workflows
- 17:57Team data pipelines
- 23:59Code review best practices
- 27:54Cross-team architecture
- 31:51Collaboration tools
- 35:05Training strategies
- 37:13Q&A
related talks
36mThe Lifecycle of a Jupyter Environment
41mBetter Code for Data Science
28mThe Coding Portion of Data Science
31mProductionalizing a Data Science Team
42mSimplifying Data Analysis with GitHub Codespaces, Jupyter Notebooks & Open AI
38mBuilding Data Pipelines in Python
01h 19mDatascience on the web
28mCollaborate with your team using data science notebooks