The 10 commandments of reliable data science

on YouTube ↗30m

" summary

Ten concrete rules: start projects with a repeatable layout (example: cookiecutter-data-science), keep raw extracts immutable, run the DAG from raw input to final artifacts automatically, and centralize all model and run parameters. Use notebooks for exploration and move reusable code into source files so it can be unit tested; add assertion tests and logging so runs fail loudly and produce saved artifacts. Practical tools mentioned include NB Auto export, Makefiles and orchestration to reproduce results.

AI-written summary · about

" chapters (11)

1. 0:00Reliable data science intro
2. 1:30Criticism of data science
3. 4:43Organize projects early
4. 7:53Version control for DS
5. 10:01Notebooks for exploration
6. 12:08Test assumptions
7. 16:24Automate project runs
8. 19:50Centralize parameters
9. 21:54Log run artifacts
10. 25:46Simplest end-to-end pipeline
11. 28:14Q&A

" related talks

• 26mThe Software Engineering Part of Data ScienceNiño R. Eclarin · PyConline AU 2020
• 31mWhy Most Data Projects Fail and How to Avoid ItJesse Anderson · GOTO Amsterdam 2023
• 20mWhy data privacy is critical for robust data management?Peter Wang · The Fifth Elephant 2019
• 20mSkrub DataOps, ProbablGael Varoquaux · dotAI 2025
• 48mBecoming a Data Head, a reflection on the trade offs and decisionAndy Terrel · PyData Austin 2019
• 42mThe 7 habits of data effective companiesLars Albertsson · Øredev 2022
• 30mDevOps For Data TeamsTobias Macey · DevOpsDays Boston 2018
• 32mWhy Most Data Projects Fail and How to Avoid ItJesse Anderson · YOW! Melbourne 2022