A Theoretical Framework for Statistical Evaluability of Generative Models
summary
Statistical evaluability is formalized: some metrics are impossible to test from finite i.i.d. samples while others are. Concrete findings: KL and Rényi are not weakly evaluable (perplexity cannot reliably recover TV or restricted KL), and simple precision/hallucination is also not evaluable. A broad family—integral probability metrics (IPMs)—is evaluable when the test class has finite VC or fat-shattering dimension; otherwise IPMs are only 3-weakly evaluable and the factor three is optimal. Proofs use density‑estimation proxies and a new covering argument that tightens uniform‑convergence constants to two.
AI-written summary · about
chapters (13)
- 0:00Statistical evaluation intro
- 2:32When to use evaluability
- 5:03Defining evaluability
- 9:01Evaluability vs loss minimization
- 11:16Evaluability of perplexity & KL
- 13:48Precision & hallucination rates
- 17:58Test-based vs IP metrics
- 23:02IPM dichotomy by VC & fat-shattering
- 30:36Lower bounds for IPM evaluability
- 36:48Uniform convergence & covering numbers
- 37:32Summary of evaluability results
- 39:38Open problems
- 40:05Q&A
related talks
11mThe Unbearable Lightness of (Agentic) Evaluations
33mThe Challenge of Valid Evaluations
59mCompositional Properties of Statistical Procedures: An Information-Theoretic View
40mMeasuring the unmeasurable: How to evaluate generative AI-based systems?
01h 15mAn overview of classical robust statistics and generalizations to the future
29mEvaluation Design
47mUnprovability and Other Impossibility Results in Machine Learning
48mThe Approximate Implication Problem For Probabilistic Graphical Models