Research Reproducibility in Embedded Learning
summary
Introduces a reproducibility methodology implemented as the TriScale framework that uses nonparametric percentile confidence intervals (Thompson-style) to turn accuracy goals into minimum sample counts; demonstrates via Jupyter/Binder notebooks and iid/autocorrelation checks. Key numbers: median needs six runs for a 95% bound, extreme percentiles jump to ~300 runs, and ultrarare tails demand ~300k samples. Provides tools to compute required runs, exclude outliers, and quantify variability scores across series so performance claims come with measurable confidence.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:20Disclaimers
- 4:15Structure
- 4:55First Block
- 5:50Example
- 11:46Replicability
- 17:23Source of variability
- 24:27Recap
- 24:57What is a statistic
- 28:07Predictive statistics
- 31:27What should we do
- 32:10Confidence Interval
- 33:04Nonparametric Statistics
- 35:02Questions
- 38:30Framework
- 49:40Launch Binder
- 50:34Jupyter Notebook
related talks
30mReproducible Machine Learning
30mReproducible Machine Learning at Scale
25mReproducibility in Embedding Benchmarks
29mReproducibility in ML Systems: A Netflix Original
25mVerifying and evaluating scientific results with the open source package "scores"
01h 16mReproducible machine learning and science with python
38mUsing Reproducible Experiments To Create Better Models
32mReproducibility in Embedding Benchmarks