Evidence-Based Rankings
summary
Defines domination compatibility and a stronger evidence consistency criterion for rankings, proves that a ranking induced by a C-multi-accurate predictor satisfies these guarantees, and reduces learning such rankings to learning multi-accurate/multi-calibrated predictors (algorithmic procedures and uniform convergence bounds over a class C). Shows standard fixed-test definitions are manipulable, requires ranking-dependent tests (quantiles of the produced ranking) and proves equivalence between ranking-reflexive evidence consistency and domination compatibility. Paper appears at FOCS.
AI-written summary · about
chapters (13)
- 0:00Intro
- 2:30Motivation Ranking is often underlying motivation for prediction e.g. college admissions predict probability of graduation in 4 years
- 3:54Task Learn to rank individuals by probability of outcome
- 8:02Goal: Learn to rank individuals according to probability of success, without observing
- 12:34Ranking from outcomes Statistical queries: From a small sample of outcomes, can estimate
- 14:06Comparison to prior work Lots of prior work on learning to rank
- 15:23Consistent Predictors Every ranking has a family of consistent predictors: Any predictor that respects the order of ranking
- 21:58Why Domination Compatibility? Violations to domination compatibility feel objectionable No consistent predictor respects evidence given by C!
- 23:47Evidence Consistency A ranking ris C-evidence-consistent if there exists a consistent predictor p such that for all sets SEC
- 26:52Learning a ranking Reduce learning a ranking to learning a predictor Induced ranking of C-multi-accurate predictor p is: C-evidence-consistent (p serves as witness)
- 33:10Are we there yet? Rankings consistent with evidence in rich class of tests No transpositions of groups defined by Predefined statistical tests may not be enough
- 38:02Weakness in evidence consistency Not just an issue with adversarial manipulation
- 41:56Interpreting the theorem The surprising strength of Domination Compatibility → When the sets in question may depend on ranking, domination compatibility implies global consistency Requires self-reference Domination-compatibility strictly weaker than evidence-consistency when sets are predefined
related talks
34mDistributional Robustness, Learning, and Empirical Likelihood
33mA Few Connections Between Optimization and Probability
53mGroups and Individuals in Fairness and Probability
01h 15mMeasures of relatedness in domain adaptation and related problems.
01h 07mDoes generalization imply accuracy on the line? A new look at robust generalization
40mOmnipredicting Single-Index Models with Multi-Index Models
10mUsing Dominance Analysis for accurate feature importance
24mLearning "Learning to Rank"