Provenance for Reproducible Data Science
summary
Claims that provenance captured with the W3C PROV model and Python libraries can make data analytics reproducible are backed by concrete work: provenance from instrumented Jupyter notebooks and TensorFlow training is emitted, stored in a graph database (Neo4j), and analysed (shortest‑path and dependency queries); a 400‑iteration training run yields on the order of 64,000 provenance nodes. The pipeline supports export formats and even blockchain anchoring for tamper evidence.
AI-written summary · about
chapters (9)
- 0:00Provenance in data science
- 5:58What is metadata and provenance
- 10:20W3C provenance model
- 16:50Storing and visualizing provenance
- 21:53Recording provenance in apps
- 25:20Live demo: provenance instrumentation
- 27:53Provenance in ML workflows
- 29:43Challenges in provenance communication
- 30:03Summary and call to action
related talks
33mTowards Interpretable Data Science
26mDebugging Large-Scale Datalog: A Scalable Provenance Evaluation Strategy
30mAchieving Understandability and Privacy With Provenance-Based Explanations
35mData versioning in machine learning projects
17mOpen Source for Data Quality
30mGraph Data Science
25mFrom SQL to Python: Building Data Context for Agents and people
46mFrictionless Data Science w/Python