An Information Theoretic Metric for Multi-Class Categorization
summary
Introduces an information-theoretic performance metric for evaluating classifiers and categorization systems that works for binary, multi-class, and multi-label settings. Describes limitations of standard metrics (accuracy, precision, recall, F1) and presents a metric intended to be dataset-comparable; implementation code is provided under the Apache open-source license.
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:45Examples
- 2:11Why Evaluate Predictors?
- 3:02Metrics Based on the Confusion Matrix
- 5:19F-Score
- 6:49Matthews correlation coefficient
- 7:13Uncertainty coefficient
- 8:30Comparison
- 10:39Predictor Re-Labeling
- 11:20Multi-Class Prediction
- 12:11Pearson's Define
- 13:28This Metric is Old! Why is it Ignored?
- 15:12Multi-Label Categorization
- 18:02Precision & Recall - ?!
- 18:45Proficiency: Definition Introduce binary random variables
- 19:21Proficiency: Permuted
- 19:37Proficiency: Properties
- 20:44Example: Brand Detection
- 22:08Numeric Stability
- 22:25Implementation
related talks
14mFair And Reliable Machine Learning For High-Stakes Applications:approaches Using Information Theory
32mSemantic Label Representation and Multimodal Categorization
37mAlgorithmic Fairness From The Lens Of Causality And Information Theory
40mMachine Learning Model Evaluation Metrics
28mClassification Through Regression:Unlock the True Potential of Your Labels
27mDetection and Estimation through an Information Theory Lens
53mGroups and Individuals in Fairness and Probability
29mEvaluation Design