Statistical Typing: A Runtime TypingSystem for Data Science&Machine Learning
summary
Proposes statistical typing as a runtime typing system that extends bool/str/float with deterministic and probabilistic constraints, implements schemas as generative data contracts, and uses pandera to validate, sample, and integrate with pandas, pytest, sklearn and hypothesis strategy generation. Shows schemas catching a bad pipeline column, bootstrapping schemas from samples, and using sampled data for property tests. Demonstrates practical CI/alerting use and admits pandera is a useful but incomplete implementation.
AI-written summary · about
chapters (8)
- 0:00Start
- 1:41What Would a Type System Geared toward Data Science and Machine Learning Look like
- 2:55How Do I Know if F Is Working as Intended
- 3:59Unit Tests
- 4:53Define Property-Based Tests
- 10:25Extending Floats into Gaussian Types
- 12:31Implications of a Fully Implemented Statistical Type System
- 18:58Generative Adversarial Networks
related talks
47mEnforcing Data Quality in Data Processing and ML Pipelines with Flyte and Pandera
52mDoubt Truth to be a Liar: Non Trivialityy of Type Safety for Machine Learning
36mProbabilistic scripts for automating common-sense tasks
29mSchemas: The Key To Data Happiness
33mThe state of Scala as a platform for statistical computing
21mFP for Data Science For Loops Considered Harmful
18mConcrete Type Inference for Code Optimization using Machine Learning with SMT S...
33mScala DSLs and Probabilistic Programming